fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Akanksha Mahajan 2db6a4a1d6 Seek parallelization (#9994) ... Summary: The RocksDB iterator is a hierarchy of iterators. MergingIterator maintains a heap of LevelIterators, one for each L0 file and for each non-zero level. The Seek() operation naturally lends itself to parallelization, as it involves positioning every LevelIterator on the correct data block in the correct SST file. It lookups a level for a target key, to find the first key that's >= the target key. This typically involves reading one data block that is likely to contain the target key, and scan forward to find the first valid key. The forward scan may read more data blocks. In order to find the right data block, the iterator may read some metadata blocks (required for opening a file and searching the index). This flow can be parallelized. Design: Seek will be called two times under async_io option. First seek will send asynchronous request to prefetch the data blocks at each level and second seek will follow the normal flow and in FilePrefetchBuffer::TryReadFromCacheAsync it will wait for the Poll() to get the results and add the iterator to min_heap. - Status::TryAgain is passed down from FilePrefetchBuffer::PrefetchAsync to block_iter_.Status indicating asynchronous request has been submitted. - If for some reason asynchronous request returns error in submitting the request, it will fallback to sequential reading of blocks in one pass. - If the data already exists in prefetch_buffer, it will return the data without prefetching further and it will be treated as single pass of seek. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9994 Test Plan: - **Run Regressions.** ``` ./db_bench -db=/tmp/prefix_scan_prefetch_main -benchmarks="fillseq" -key_size=32 -value_size=512 -num=5000000 -use_direct_io_for_flush_and_compaction=true -target_file_size_base=16777216 ``` i) Previous release 7.0 run for normal prefetching with async_io disabled: ``` ./db_bench -use_existing_db=true -db=/tmp/prefix_scan_prefetch_main -benchmarks="seekrandom" -key_size=32 -value_size=512 -num=5000000 -use_direct_reads=true -seek_nexts=327680 -duration=120 -ops_between_duration_checks=1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags RocksDB: version 7.0 Date: Thu Mar 17 13:11:34 2022 CPU: 24 * Intel Core Processor (Broadwell) CPUCache: 16384 KB Keys: 32 bytes each (+ 0 bytes user-defined timestamp) Values: 512 bytes each (256 bytes after compression) Entries: 5000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 2594.0 MB (estimated) FileSize: 1373.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: Snappy Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/tmp/prefix_scan_prefetch_main] seekrandom : 483618.390 micros/op 2 ops/sec; 338.9 MB/s (249 of 249 found) ``` ii) normal prefetching after changes with async_io disable: ``` ./db_bench -use_existing_db=true -db=/tmp/prefix_scan_prefetch_main -benchmarks="seekrandom" -key_size=32 -value_size=512 -num=5000000 -use_direct_reads=true -seek_nexts=327680 -duration=120 -ops_between_duration_checks=1 Set seed to 1652922591315307 because --seed was 0 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags RocksDB: version 7.3 Date: Wed May 18 18:09:51 2022 CPU: 32 * Intel Xeon Processor (Skylake) CPUCache: 16384 KB Keys: 32 bytes each (+ 0 bytes user-defined timestamp) Values: 512 bytes each (256 bytes after compression) Entries: 5000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 2594.0 MB (estimated) FileSize: 1373.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: Snappy Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/tmp/prefix_scan_prefetch_main] seekrandom : 483080.466 micros/op 2 ops/sec 120.287 seconds 249 operations; 340.8 MB/s (249 of 249 found) ``` iii) db_bench with async_io enabled completed succesfully ``` ./db_bench -use_existing_db=true -db=/tmp/prefix_scan_prefetch_main -benchmarks="seekrandom" -key_size=32 -value_size=512 -num=5000000 -use_direct_reads=true -seek_nexts=327680 -duration=120 -ops_between_duration_checks=1 -async_io=1 -adaptive_readahead=1 Set seed to 1652924062021732 because --seed was 0 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags RocksDB: version 7.3 Date: Wed May 18 18:34:22 2022 CPU: 32 * Intel Xeon Processor (Skylake) CPUCache: 16384 KB Keys: 32 bytes each (+ 0 bytes user-defined timestamp) Values: 512 bytes each (256 bytes after compression) Entries: 5000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 2594.0 MB (estimated) FileSize: 1373.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: Snappy Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/tmp/prefix_scan_prefetch_main] seekrandom : 553913.576 micros/op 1 ops/sec 120.199 seconds 217 operations; 293.6 MB/s (217 of 217 found) ``` - db_stress with async_io disabled completed succesfully ``` export CRASH_TEST_EXT_ARGS=" --async_io=0" make crash_test -j ``` I**n Progress**: db_stress with async_io is failing and working on debugging/fixing it. Reviewed By: anand1976 Differential Revision: D36459323 Pulled By: akankshamahajan15 fbshipit-source-id: abb1cd944abe712bae3986ae5b16704b3338917c		3 years ago
..
binary_search_index_reader.cc	Set Read rate limiter priority dynamically and pass it to FS (#9996)	3 years ago
binary_search_index_reader.h	Extend Get/MultiGet deadline support to table open (#6982)	5 years ago
block.cc	Use std::numeric_limits<> (#9954)	3 years ago
block.h	Reduce comparator objects init cost in BlockIter (#9611)	3 years ago
block_based_filter_block.cc	Remove own ToString() (#9955)	3 years ago
block_based_filter_block.h	FilterPolicy API changes for 7.0 (#9501)	3 years ago
block_based_filter_block_test.cc	FilterPolicy API changes for 7.0 (#9501)	3 years ago
block_based_table_builder.cc	Support using ZDICT_finalizeDictionary to generate zstd dictionary (#9857)	3 years ago
block_based_table_builder.h	Fix segmentation fault in table_options.prepopulate_block_cache when used with partition_filters (#9263)	3 years ago
block_based_table_factory.cc	Mark old reserve* option deprecated (#10016)	3 years ago
block_based_table_factory.h	Account memory of big memory users in BlockBasedTable in global memory limit (#9748)	3 years ago
block_based_table_iterator.cc	Seek parallelization (#9994)	3 years ago
block_based_table_iterator.h	Seek parallelization (#9994)	3 years ago
block_based_table_reader.cc	Seek parallelization (#9994)	3 years ago
block_based_table_reader.h	Seek parallelization (#9994)	3 years ago
block_based_table_reader_impl.h	Seek parallelization (#9994)	3 years ago
block_based_table_reader_sync_and_async.h	Seek parallelization (#9994)	3 years ago
block_based_table_reader_test.cc	Multi file concurrency in MultiGet using coroutines and async IO (#9968)	3 years ago
block_builder.cc	Improve data block construction performance (#9040)	3 years ago
block_builder.h	Improve data block construction performance (#9040)	3 years ago
block_like_traits.h	Change type of cache buffer passed to `Cache::CreateCallback()` to `const void*` (#9595)	3 years ago
block_prefetcher.cc	Seek parallelization (#9994)	3 years ago
block_prefetcher.h	Set Read rate limiter priority dynamically and pass it to FS (#9996)	3 years ago
block_prefix_index.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
block_prefix_index.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
block_test.cc	Improve / clean up meta block code & integrity (#9163)	3 years ago
block_type.h	Fix and detect headers with missing dependencies (#8893)	4 years ago
cachable_entry.h	Meta-internal folly integration with F14FastMap (#9546)	3 years ago
data_block_footer.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
data_block_footer.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
data_block_hash_index.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
data_block_hash_index.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
data_block_hash_index_test.cc	Fast path for detecting unchanged prefix_extractor (#9407)	3 years ago
filter_block.h	Fix auto_prefix_mode performance with partitioned filters (#10012)	3 years ago
filter_block_reader_common.cc	Seek parallelization (#9994)	3 years ago
filter_block_reader_common.h	Fix auto_prefix_mode performance with partitioned filters (#10012)	3 years ago
filter_policy.cc	Rewrite memory-charging feature's option API (#9926)	3 years ago
filter_policy_internal.h	Fix a major performance bug in 7.0 re: filter compatibility (#9736)	3 years ago
flush_block_policy.cc	Restore Regex support for ObjectLibrary::Register, rename new APIs to allow old one to be deprecated in the future (#9362)	3 years ago
flush_block_policy.h	Make FlushBlockPolicyFactory into a Customizable class (#8432)	4 years ago
full_filter_block.cc	Fix auto_prefix_mode performance with partitioned filters (#10012)	3 years ago
full_filter_block.h	Fix auto_prefix_mode performance with partitioned filters (#10012)	3 years ago
full_filter_block_test.cc	Fix a major performance bug in 7.0 re: filter compatibility (#9736)	3 years ago
hash_index_reader.cc	Set Read rate limiter priority dynamically and pass it to FS (#9996)	3 years ago
hash_index_reader.h	Extend Get/MultiGet deadline support to table open (#6982)	5 years ago
index_builder.cc	Add (& fix) some simple source code checks (#8821)	4 years ago
index_builder.h	Remove own ToString() (#9955)	3 years ago
index_reader_common.cc	Seek parallelization (#9994)	3 years ago
index_reader_common.h	Set Read rate limiter priority dynamically and pass it to FS (#9996)	3 years ago
mock_block_based_table.h	Refactor FilterPolicies toward Customizable (#9567)	3 years ago
parsed_full_filter_block.cc	Hide FilterBits{Builder,Reader} from public API (#9592)	3 years ago
parsed_full_filter_block.h	Use new Insert and Lookup APIs in table reader to support secondary cache (#8315)	4 years ago
partitioned_filter_block.cc	Seek parallelization (#9994)	3 years ago
partitioned_filter_block.h	Meta-internal folly integration with F14FastMap (#9546)	3 years ago
partitioned_filter_block_test.cc	More refactoring ahead of footer & meta changes (#9240)	3 years ago
partitioned_index_iterator.cc	Seek parallelization (#9994)	3 years ago
partitioned_index_iterator.h	Make initial auto readahead_size configurable (#9836)	3 years ago
partitioned_index_reader.cc	Seek parallelization (#9994)	3 years ago
partitioned_index_reader.h	Meta-internal folly integration with F14FastMap (#9546)	3 years ago
reader_common.cc	Remove own ToString() (#9955)	3 years ago
reader_common.h	Bring the Configurable options together (#5753)	5 years ago
uncompression_dict_reader.cc	Seek parallelization (#9994)	3 years ago
uncompression_dict_reader.h	Fb 9718 verify checksums is ignored (#9767)	3 years ago