fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Changyu Bi 6a0f637633 Compare the number of input keys and processed keys for compactions (#11571) ... Summary: ... to improve data integrity validation during compaction. A new option `compaction_verify_record_count` is introduced for this verification and is enabled by default. One exception when the verification is not done is when a compaction filter returns kRemoveAndSkipUntil which can cause CompactionIterator to seek until some key and hence not able to keep track of the number of keys processed. For expected number of input keys, we sum over the number of total keys - number of range tombstones across compaction input files (`CompactionJob::UpdateCompactionStats()`). Table properties are consulted if `FileMetaData` is not initialized for some input file. Since table properties for all input files were also constructed during `DBImpl::NotifyOnCompactionBegin()`, `Compaction::GetTableProperties()` is introduced to reduce duplicated code. For actual number of keys processed, each subcompaction will record its number of keys processed to `sub_compact->compaction_job_stats.num_input_records` and aggregated when all subcompactions finish (`CompactionJob::AggregateCompactionStats()`). In the case when some subcompaction encountered kRemoveAndSkipUntil from compaction filter and does not have accurate count, it propagates this information through `sub_compact->compaction_job_stats.has_num_input_records`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11571 Test Plan: * Add a new unit test `DBCompactionTest.VerifyRecordCount` for the corruption case. * All other unit tests for non-corrupted case. * Ran crash test for a few hours: `python3 ./tools/db_crashtest.py whitebox --simple` Reviewed By: ajkr Differential Revision: D47131965 Pulled By: cbi42 fbshipit-source-id: cc8e94565dd526c4347e9d3843ecf32f6727af92		1 year ago
..
clipping_iterator.h	Refactor AddRangeDels() + consider range tombstone during compaction file cutting (#11113)	2 years ago
clipping_iterator_test.cc	Print stack traces on frozen tests in CI (#10828)	2 years ago
compaction.cc	Compare the number of input keys and processed keys for compactions (#11571)	1 year ago
compaction.h	Compare the number of input keys and processed keys for compactions (#11571)	1 year ago
compaction_iteration_stats.h	Support readahead during compaction for blob files (#9187)	3 years ago
compaction_iterator.cc	Compare the number of input keys and processed keys for compactions (#11571)	1 year ago
compaction_iterator.h	Compare the number of input keys and processed keys for compactions (#11571)	1 year ago
compaction_iterator_test.cc	Compare the number of input keys and processed keys for compactions (#11571)	1 year ago
compaction_job.cc	Compare the number of input keys and processed keys for compactions (#11571)	1 year ago
compaction_job.h	Compare the number of input keys and processed keys for compactions (#11571)	1 year ago
compaction_job_stats_test.cc	Make option `level_compaction_dynamic_level_bytes` true by default (#11525)	2 years ago
compaction_job_test.cc	Compare the number of input keys and processed keys for compactions (#11571)	1 year ago
compaction_outputs.cc	Optimize about sstableKeyCompare (#11610)	1 year ago
compaction_outputs.h	fix: std::optional value() build error on older macOS SDK (#11574)	1 year ago
compaction_picker.cc	Do not include last level in compaction when `allow_ingest_behind=true` (#11489)	2 years ago
compaction_picker.h	Remove RocksDB LITE (#11147)	2 years ago
compaction_picker_fifo.cc	Support compacting files to different temperatures in FIFO compaction (#11428)	2 years ago
compaction_picker_fifo.h	Support compacting files to different temperatures in FIFO compaction (#11428)	2 years ago
compaction_picker_level.cc	Try to pick more files in `LevelCompactionBuilder::TryExtendNonL0TrivialMove()` (#11347)	2 years ago
compaction_picker_level.h	Sort L0 files by newly introduced epoch_num (#10922)	2 years ago
compaction_picker_test.cc	Record the `persist_user_defined_timestamps` flag in manifest (#11515)	2 years ago
compaction_picker_universal.cc	Do not include last level in compaction when `allow_ingest_behind=true` (#11489)	2 years ago
compaction_picker_universal.h	Remove RocksDB LITE (#11147)	2 years ago
compaction_service_job.cc	Remove RocksDB LITE (#11147)	2 years ago
compaction_service_test.cc	Remove wait_unscheduled from waitForCompact internal API (#11443)	2 years ago
compaction_state.cc	Tiered Compaction: per key placement support (#9964)	2 years ago
compaction_state.h	Tiered Compaction: per key placement support (#9964)	2 years ago
file_pri.h	Try to start TTL earlier with kMinOverlappingRatio is used (#8749)	3 years ago
sst_partitioner.cc	Remove FactoryFunc from LoadXXXObject (#11203)	2 years ago
subcompaction_state.cc	Refactor Compaction file cut `ShouldStopBefore()` (#10629)	2 years ago
subcompaction_state.h	Refactor AddRangeDels() + consider range tombstone during compaction file cutting (#11113)	2 years ago
tiered_compaction_test.cc	Add WaitForCompact with WaitForCompactOptions to public API (#11436)	2 years ago