fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Changyu Bi f02c708aa3 Consider range tombstone in compaction output file cutting (#10802) ... Summary: This PR is the first step for Issue https://github.com/facebook/rocksdb/issues/4811. Currently compaction output files are cut at point keys, and the decision is made mainly in `CompactionOutputs::ShouldStopBefore()`. This makes it possible for range tombstones to cause large compactions that does not respect `max_compaction_bytes`. For example, we can have a large range tombstone that overlaps with too many files from the next level. Another example is when there is a gap between a range tombstone and another key. The first issue may be more acceptable, as a lot of data is deleted. This PR address the second issue by calling `ShouldStopBefore()` for range tombstone start keys. The main change is for `CompactionIterator` to emit range tombstone start keys to be processed by `CompactionOutputs`. A new `CompactionMergingIterator` is introduced and only used under `CompactionIterator` for this purpose. Further improvement after this PR include 1) cut compaction output at some grandparent boundary key instead of at the next point key or range tombstone start key and 2) cut compaction output file within a large range tombstone (it may be easier and reasonable to only do it for range tombstones at the end of a compaction output). Pull Request resolved: https://github.com/facebook/rocksdb/pull/10802 Test Plan: - added unit tests in db_range_del_test. - stress test: `python3 tools/db_crashtest.py whitebox --[simple|enable_ts] --verify_iterator_with_expected_state_one_in=5 --delrangepercent=5 --prefixpercent=2 --writepercent=58 --readpercen=21 --duration=36000 --range_deletion_width=1000000` Reviewed By: ajkr, jay-zhuang Differential Revision: D40308827 Pulled By: cbi42 fbshipit-source-id: a8fd6f70a3f09d0ef7a40e006f6c964bba8c00df		2 years ago
..
blob_constants.h	Move BlobDB related files under db/ to db/blob/ (#6519)	5 years ago
blob_contents.cc	Support custom allocators for the blob cache (#10628)	2 years ago
blob_contents.h	Support custom allocators for the blob cache (#10628)	2 years ago
blob_counting_iterator.h	Consider range tombstone in compaction output file cutting (#10802)	2 years ago
blob_counting_iterator_test.cc	Print stack traces on frozen tests in CI (#10828)	2 years ago
blob_fetcher.cc	Support readahead during compaction for blob files (#9187)	3 years ago
blob_fetcher.h	Support readahead during compaction for blob files (#9187)	3 years ago
blob_file_addition.cc	Print blob file checksums as hex (#8437)	3 years ago
blob_file_addition.h	Move BlobDB related files under db/ to db/blob/ (#6519)	5 years ago
blob_file_addition_test.cc	Print stack traces on frozen tests in CI (#10828)	2 years ago
blob_file_builder.cc	Support custom allocators for the blob cache (#10628)	2 years ago
blob_file_builder.h	Support prepopulating/warming the blob cache (#10298)	2 years ago
blob_file_builder_test.cc	Print stack traces on frozen tests in CI (#10828)	2 years ago
blob_file_cache.cc	Rename ImmutableOptions variables (#8409)	3 years ago
blob_file_cache.h	Rename ImmutableOptions variables (#8409)	3 years ago
blob_file_cache_test.cc	Print stack traces on frozen tests in CI (#10828)	2 years ago
blob_file_completion_callback.h	Fix LITE mode builds on MacOs (#8981)	3 years ago
blob_file_garbage.cc	Move BlobDB related files under db/ to db/blob/ (#6519)	5 years ago
blob_file_garbage.h	Move BlobDB related files under db/ to db/blob/ (#6519)	5 years ago
blob_file_garbage_test.cc	Print stack traces on frozen tests in CI (#10828)	2 years ago
blob_file_meta.cc	Add BlobMetaData retrieval methods (#8273)	3 years ago
blob_file_meta.h	Add BlobMetaData retrieval methods (#8273)	3 years ago
blob_file_reader.cc	Eliminate some allocations/copies around the blob cache (#10647)	2 years ago
blob_file_reader.h	Eliminate some allocations/copies around the blob cache (#10647)	2 years ago
blob_file_reader_test.cc	Print stack traces on frozen tests in CI (#10828)	2 years ago
blob_garbage_meter.cc	Add a class for measuring the amount of garbage generated during compaction (#8426)	3 years ago
blob_garbage_meter.h	Add a class for measuring the amount of garbage generated during compaction (#8426)	3 years ago
blob_garbage_meter_test.cc	Print stack traces on frozen tests in CI (#10828)	2 years ago
blob_index.h	Remove local static string (#8103)	2 years ago
blob_log_format.cc	Remove local static string (#8103)	2 years ago
blob_log_format.h	Run clang-format on db/blob/ (#10856)	2 years ago
blob_log_sequential_reader.cc	Add rate limiter priority to ReadOptions (#9424)	3 years ago
blob_log_sequential_reader.h	Fix a issue with initializing blob header buffer (#8537)	3 years ago
blob_log_writer.cc	WritableFileWriter tries to skip operations after failure (#10489)	2 years ago
blob_log_writer.h	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033)	4 years ago
blob_read_request.h	Enable blob caching for MultiGetBlob in RocksDB (#10272)	2 years ago
blob_source.cc	Eliminate some allocations/copies around the blob cache (#10647)	2 years ago
blob_source.h	Eliminate some allocations/copies around the blob cache (#10647)	2 years ago
blob_source_test.cc	Print stack traces on frozen tests in CI (#10828)	2 years ago
db_blob_basic_test.cc	Run clang-format on db/blob/ (#10856)	2 years ago
db_blob_compaction_test.cc	Fix wrong value passed in compaction filter in BlobDB (#10391)	2 years ago
db_blob_corruption_test.cc	Enable a few unit tests to use custom Env objects (#9087)	3 years ago
db_blob_index_test.cc	Remove own ToString() (#9955)	3 years ago
prefetch_buffer_collection.cc	Support readahead during compaction for blob files (#9187)	3 years ago
prefetch_buffer_collection.h	Support readahead during compaction for blob files (#9187)	3 years ago