fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Changyu Bi 229297d1b8 Refactor AddRangeDels() + consider range tombstone during compaction file cutting (#11113) ... Summary: A second attempt after https://github.com/facebook/rocksdb/issues/10802, with bug fixes and refactoring. This PR updates compaction logic to take range tombstones into account when determining whether to cut the current compaction output file (https://github.com/facebook/rocksdb/issues/4811). Before this change, only point keys were considered, and range tombstones could cause large compactions. For example, if the current compaction outputs is a range tombstone [a, b) and 2 point keys y, z, they would be added to the same file, and may overlap with too many files in the next level and cause a large compaction in the future. This PR also includes ajkr's effort to simplify the logic to add range tombstones to compaction output files in `AddRangeDels()` ([https://github.com/facebook/rocksdb/issues/11078](https://github.com/facebook/rocksdb/pull/11078#issuecomment-1386078861)). The main change is for `CompactionIterator` to emit range tombstone start keys to be processed by `CompactionOutputs`. A new class `CompactionMergingIterator` is introduced to replace `MergingIterator` under `CompactionIterator` to enable emitting of range tombstone start keys. Further improvement after this PR include cutting compaction output at some grandparent boundary key (instead of the next output key) when cutting within a range tombstone to reduce overlap with grandparents. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11113 Test Plan: * added unit test in db_range_del_test * crash test with a small key range: `python3 tools/db_crashtest.py blackbox --simple --max_key=100 --interval=600 --write_buffer_size=262144 --target_file_size_base=256 --max_bytes_for_level_base=262144 --block_size=128 --value_size_mult=33 --subcompactions=10 --use_multiget=1 --delpercent=3 --delrangepercent=2 --verify_iterator_with_expected_state_one_in=2 --num_iterations=10` Reviewed By: ajkr Differential Revision: D42655709 Pulled By: cbi42 fbshipit-source-id: 8367e36ef5640e8f21c14a3855d4a8d6e360a34c		2 years ago
..
clipping_iterator.h	Refactor AddRangeDels() + consider range tombstone during compaction file cutting (#11113)	2 years ago
clipping_iterator_test.cc	Print stack traces on frozen tests in CI (#10828)	2 years ago
compaction.cc	Revert #10802 Consider range tombstone in compaction output file cutting (#11089)	2 years ago
compaction.h	Revert #10802 Consider range tombstone in compaction output file cutting (#11089)	2 years ago
compaction_iteration_stats.h	Support readahead during compaction for blob files (#9187)	3 years ago
compaction_iterator.cc	Refactor AddRangeDels() + consider range tombstone during compaction file cutting (#11113)	2 years ago
compaction_iterator.h	Refactor AddRangeDels() + consider range tombstone during compaction file cutting (#11113)	2 years ago
compaction_iterator_test.cc	Basic Support for Merge with user-defined timestamp (#10819)	2 years ago
compaction_job.cc	Refactor AddRangeDels() + consider range tombstone during compaction file cutting (#11113)	2 years ago
compaction_job.h	Refactor AddRangeDels() + consider range tombstone during compaction file cutting (#11113)	2 years ago
compaction_job_stats_test.cc	Remove RocksDB LITE (#11147)	2 years ago
compaction_job_test.cc	Remove RocksDB LITE (#11147)	2 years ago
compaction_outputs.cc	Refactor AddRangeDels() + consider range tombstone during compaction file cutting (#11113)	2 years ago
compaction_outputs.h	Refactor AddRangeDels() + consider range tombstone during compaction file cutting (#11113)	2 years ago
compaction_picker.cc	Remove RocksDB LITE (#11147)	2 years ago
compaction_picker.h	Remove RocksDB LITE (#11147)	2 years ago
compaction_picker_fifo.cc	Remove RocksDB LITE (#11147)	2 years ago
compaction_picker_fifo.h	Remove RocksDB LITE (#11147)	2 years ago
compaction_picker_level.cc	Add missing range conflict check between file ingestion and RefitLevel() (#10988)	2 years ago
compaction_picker_level.h	Sort L0 files by newly introduced epoch_num (#10922)	2 years ago
compaction_picker_test.cc	Remove RocksDB LITE (#11147)	2 years ago
compaction_picker_universal.cc	Remove RocksDB LITE (#11147)	2 years ago
compaction_picker_universal.h	Remove RocksDB LITE (#11147)	2 years ago
compaction_service_job.cc	Remove RocksDB LITE (#11147)	2 years ago
compaction_service_test.cc	Remove RocksDB LITE (#11147)	2 years ago
compaction_state.cc	Tiered Compaction: per key placement support (#9964)	2 years ago
compaction_state.h	Tiered Compaction: per key placement support (#9964)	2 years ago
file_pri.h	Try to start TTL earlier with kMinOverlappingRatio is used (#8749)	3 years ago
sst_partitioner.cc	Remove FactoryFunc from LoadXXXObject (#11203)	2 years ago
subcompaction_state.cc	Refactor Compaction file cut `ShouldStopBefore()` (#10629)	2 years ago
subcompaction_state.h	Refactor AddRangeDels() + consider range tombstone during compaction file cutting (#11113)	2 years ago
tiered_compaction_test.cc	Remove RocksDB LITE (#11147)	2 years ago