fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Changyu Bi 9d77bf8f7b Fragment memtable range tombstone in the write path (#10380) ... Summary: - Right now each read fragments the memtable range tombstones https://github.com/facebook/rocksdb/issues/4808. This PR explores the idea of fragmenting memtable range tombstones in the write path and reads can just read this cached fragmented tombstone without any fragmenting cost. This PR only does the caching for immutable memtable, and does so right before a memtable is added to an immutable memtable list. The fragmentation is done without holding mutex to minimize its performance impact. - db_bench is updated to print out the number of range deletions executed if there is any. Pull Request resolved: https://github.com/facebook/rocksdb/pull/10380 Test Plan: - CI, added asserts in various places to check whether a fragmented range tombstone list should have been constructed. - Benchmark: as this PR only optimizes immutable memtable path, the number of writes in the benchmark is chosen such an immutable memtable is created and range tombstones are in that memtable. ``` single thread: ./db_bench --benchmarks=fillrandom,readrandom --writes_per_range_tombstone=1 --max_write_buffer_number=100 --min_write_buffer_number_to_merge=100 --writes=500000 --reads=100000 --max_num_range_tombstones=100 multi_thread ./db_bench --benchmarks=fillrandom,readrandom --writes_per_range_tombstone=1 --max_write_buffer_number=100 --min_write_buffer_number_to_merge=100 --writes=15000 --reads=20000 --threads=32 --max_num_range_tombstones=100 ``` Commit 99cdf16464a057ca44de2f747541dedf651bae9e is included in benchmark result. It was an earlier attempt where tombstones are fragmented for each write operation. Reader threads share it using a shared_ptr which would slow down multi-thread read performance as seen in benchmark results. Results are averaged over 5 runs. Single thread result: | Max # tombstones | main fillrandom micros/op | 99cdf16464a057ca44de2f747541dedf651bae9e | Post PR | main readrandom micros/op | 99cdf16464a057ca44de2f747541dedf651bae9e | Post PR | | ------------- | ------------- |------------- |------------- |------------- |------------- |------------- | | 0 |6.68 |6.57 |6.72 |4.72 |4.79 |4.54 | | 1 |6.67 |6.58 |6.62 |5.41 |4.74 |4.72 | | 10 |6.59 |6.5 |6.56 |7.83 |4.69 |4.59 | | 100 |6.62 |6.75 |6.58 |29.57 |5.04 |5.09 | | 1000 |6.54 |6.82 |6.61 |320.33 |5.22 |5.21 | 32-thread result: note that "Max # tombstones" is per thread. | Max # tombstones | main fillrandom micros/op | 99cdf16464a057ca44de2f747541dedf651bae9e | Post PR | main readrandom micros/op | 99cdf16464a057ca44de2f747541dedf651bae9e | Post PR | | ------------- | ------------- |------------- |------------- |------------- |------------- |------------- | | 0 |234.52 |260.25 |239.42 |5.06 |5.38 |5.09 | | 1 |236.46 |262.0 |231.1 |19.57 |22.14 |5.45 | | 10 |236.95 |263.84 |251.49 |151.73 |21.61 |5.73 | | 100 |268.16 |296.8 |280.13 |2308.52 |22.27 |6.57 | Reviewed By: ajkr Differential Revision: D37916564 Pulled By: cbi42 fbshipit-source-id: 05d6d2e16df26c374c57ddcca13a5bfe9d5b731e		2 years ago
..
adaptive	More refactoring ahead of footer & meta changes (#9240)	3 years ago
block_based	Break TableReader MultiGet into filter and lookup stages (#10432)	2 years ago
cuckoo	Remove own ToString() (#9955)	3 years ago
plain	Add seqno to time mapping (#10338)	2 years ago
block_fetcher.cc	Provide support for direct_reads with async_io (#10197)	2 years ago
block_fetcher.h	Seek parallelization (#9994)	3 years ago
block_fetcher_test.cc	Make MemoryAllocator into a Customizable class (#8980)	3 years ago
cleanable_test.cc	Eliminate unnecessary (slow) block cache Ref()ing in MultiGet (#9899)	3 years ago
format.cc	Set Read rate limiter priority dynamically and pass it to FS (#9996)	3 years ago
format.h	Remove code that only compiles for Visual Studio versions older than 2015 (#10065)	3 years ago
get_context.cc	Have Cache use Status::MemoryLimit (#10262)	2 years ago
get_context.h	Add API for writing wide-column entities (#10242)	3 years ago
internal_iterator.h	Make InternalKeyComparator not configurable (#10342)	2 years ago
iter_heap.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
iterator.cc	Eliminate unnecessary (slow) block cache Ref()ing in MultiGet (#9899)	3 years ago
iterator_wrapper.h	Reuse internal auto readhead_size at each Level (expect L0) for Iterations (#9056)	3 years ago
merger_test.cc	Cleanup multiple implementations of VectorIterator (#8901)	3 years ago
merging_iterator.cc	Seek parallelization (#9994)	3 years ago
merging_iterator.h	Cleanup includes in dbformat.h (#8930)	3 years ago
meta_blocks.cc	Add seqno to time mapping (#10338)	2 years ago
meta_blocks.h	Tests for filter compatibility (#9773)	3 years ago
mock_table.cc	Add rate limiter priority to ReadOptions (#9424)	3 years ago
mock_table.h	Fix some minor issues in the Customizable infrastructure (#8566)	3 years ago
multiget_context.h	Point-lookup returns timestamps of Delete and SingleDelete (#10056)	3 years ago
persistent_cache_helper.cc	New stable, fixed-length cache keys (#9126)	3 years ago
persistent_cache_helper.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
persistent_cache_options.h	Use STATIC_AVOID_DESTRUCTION for static objects with non-trivial destructors (#9958)	3 years ago
scoped_arena_iterator.h	Remove code that only compiles for Visual Studio versions older than 2015 (#10065)	3 years ago
sst_file_dumper.cc	Remove customized naming from InternalKeyComparator (#10343)	2 years ago
sst_file_dumper.h	Support using ZDICT_finalizeDictionary to generate zstd dictionary (#9857)	3 years ago
sst_file_reader.cc	Fast path for detecting unchanged prefix_extractor (#9407)	3 years ago
sst_file_reader_test.cc	Use the comparator from the sst file table properties in sst_dump_tool (#9491)	3 years ago
sst_file_writer.cc	Add seqno to time mapping (#10338)	2 years ago
sst_file_writer_collectors.h	Remove own ToString() (#9955)	3 years ago
table_builder.h	Add seqno to time mapping (#10338)	2 years ago
table_factory.cc	Restore Regex support for ObjectLibrary::Register, rename new APIs to allow old one to be deprecated in the future (#9362)	3 years ago
table_properties.cc	Add seqno to time mapping (#10338)	2 years ago
table_properties_internal.h	Improve / clean up meta block code & integrity (#9163)	3 years ago
table_reader.h	Break TableReader MultiGet into filter and lookup stages (#10432)	2 years ago
table_reader_bench.cc	Fast path for detecting unchanged prefix_extractor (#9407)	3 years ago
table_reader_caller.h	Fix and detect headers with missing dependencies (#8893)	3 years ago
table_test.cc	Fragment memtable range tombstone in the write path (#10380)	2 years ago
two_level_iterator.cc	Clarify caching behavior for index and filter partitions (#9068)	3 years ago
two_level_iterator.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
unique_id.cc	Track SST unique id in MANIFEST and verify (#9990)	3 years ago
unique_id_impl.h	Track SST unique id in MANIFEST and verify (#9990)	3 years ago