fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Akanksha Mahajan 2db6a4a1d6 Seek parallelization (#9994) ... Summary: The RocksDB iterator is a hierarchy of iterators. MergingIterator maintains a heap of LevelIterators, one for each L0 file and for each non-zero level. The Seek() operation naturally lends itself to parallelization, as it involves positioning every LevelIterator on the correct data block in the correct SST file. It lookups a level for a target key, to find the first key that's >= the target key. This typically involves reading one data block that is likely to contain the target key, and scan forward to find the first valid key. The forward scan may read more data blocks. In order to find the right data block, the iterator may read some metadata blocks (required for opening a file and searching the index). This flow can be parallelized. Design: Seek will be called two times under async_io option. First seek will send asynchronous request to prefetch the data blocks at each level and second seek will follow the normal flow and in FilePrefetchBuffer::TryReadFromCacheAsync it will wait for the Poll() to get the results and add the iterator to min_heap. - Status::TryAgain is passed down from FilePrefetchBuffer::PrefetchAsync to block_iter_.Status indicating asynchronous request has been submitted. - If for some reason asynchronous request returns error in submitting the request, it will fallback to sequential reading of blocks in one pass. - If the data already exists in prefetch_buffer, it will return the data without prefetching further and it will be treated as single pass of seek. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9994 Test Plan: - **Run Regressions.** ``` ./db_bench -db=/tmp/prefix_scan_prefetch_main -benchmarks="fillseq" -key_size=32 -value_size=512 -num=5000000 -use_direct_io_for_flush_and_compaction=true -target_file_size_base=16777216 ``` i) Previous release 7.0 run for normal prefetching with async_io disabled: ``` ./db_bench -use_existing_db=true -db=/tmp/prefix_scan_prefetch_main -benchmarks="seekrandom" -key_size=32 -value_size=512 -num=5000000 -use_direct_reads=true -seek_nexts=327680 -duration=120 -ops_between_duration_checks=1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags RocksDB: version 7.0 Date: Thu Mar 17 13:11:34 2022 CPU: 24 * Intel Core Processor (Broadwell) CPUCache: 16384 KB Keys: 32 bytes each (+ 0 bytes user-defined timestamp) Values: 512 bytes each (256 bytes after compression) Entries: 5000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 2594.0 MB (estimated) FileSize: 1373.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: Snappy Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/tmp/prefix_scan_prefetch_main] seekrandom : 483618.390 micros/op 2 ops/sec; 338.9 MB/s (249 of 249 found) ``` ii) normal prefetching after changes with async_io disable: ``` ./db_bench -use_existing_db=true -db=/tmp/prefix_scan_prefetch_main -benchmarks="seekrandom" -key_size=32 -value_size=512 -num=5000000 -use_direct_reads=true -seek_nexts=327680 -duration=120 -ops_between_duration_checks=1 Set seed to 1652922591315307 because --seed was 0 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags RocksDB: version 7.3 Date: Wed May 18 18:09:51 2022 CPU: 32 * Intel Xeon Processor (Skylake) CPUCache: 16384 KB Keys: 32 bytes each (+ 0 bytes user-defined timestamp) Values: 512 bytes each (256 bytes after compression) Entries: 5000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 2594.0 MB (estimated) FileSize: 1373.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: Snappy Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/tmp/prefix_scan_prefetch_main] seekrandom : 483080.466 micros/op 2 ops/sec 120.287 seconds 249 operations; 340.8 MB/s (249 of 249 found) ``` iii) db_bench with async_io enabled completed succesfully ``` ./db_bench -use_existing_db=true -db=/tmp/prefix_scan_prefetch_main -benchmarks="seekrandom" -key_size=32 -value_size=512 -num=5000000 -use_direct_reads=true -seek_nexts=327680 -duration=120 -ops_between_duration_checks=1 -async_io=1 -adaptive_readahead=1 Set seed to 1652924062021732 because --seed was 0 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags RocksDB: version 7.3 Date: Wed May 18 18:34:22 2022 CPU: 32 * Intel Xeon Processor (Skylake) CPUCache: 16384 KB Keys: 32 bytes each (+ 0 bytes user-defined timestamp) Values: 512 bytes each (256 bytes after compression) Entries: 5000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 2594.0 MB (estimated) FileSize: 1373.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: Snappy Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/tmp/prefix_scan_prefetch_main] seekrandom : 553913.576 micros/op 1 ops/sec 120.199 seconds 217 operations; 293.6 MB/s (217 of 217 found) ``` - db_stress with async_io disabled completed succesfully ``` export CRASH_TEST_EXT_ARGS=" --async_io=0" make crash_test -j ``` I**n Progress**: db_stress with async_io is failing and working on debugging/fixing it. Reviewed By: anand1976 Differential Revision: D36459323 Pulled By: akankshamahajan15 fbshipit-source-id: abb1cd944abe712bae3986ae5b16704b3338917c		3 years ago
..
utilities	Track SST unique id in MANIFEST and verify (#9990)	3 years ago
advanced_options.h	Support using ZDICT_finalizeDictionary to generate zstd dictionary (#9857)	3 years ago
c.h	Seek parallelization (#9994)	3 years ago
cache.h	Rewrite memory-charging feature's option API (#9926)	3 years ago
cache_bench_tool.h	Allow cache_bench/db_bench to use a custom secondary cache (#8312)	4 years ago
cleanable.h	Eliminate unnecessary (slow) block cache Ref()ing in MultiGet (#9899)	3 years ago
compaction_filter.h	Rename kRemoveWithSingleDelete to kPurge (#9951)	3 years ago
compaction_job_stats.h	Update compaction statistics to include the amount of data read from blob files (#8022)	4 years ago
comparator.h	Mark destructors as override (#9404)	3 years ago
compression_type.h	Move CompressionType to its own header file (#7162)	4 years ago
concurrent_task_limiter.h	Some API clarifications (#9080)	3 years ago
configurable.h	Improve performance of SliceTransform::AsString (#9401)	3 years ago
convenience.h	Specify largest_seqno in VerifyChecksum (#9919)	3 years ago
customizable.h	Mark destructors as override (#9404)	3 years ago
data_structure.h	Add (Live)FileStorageInfo API (#8968)	3 years ago
db.h	Mark GetLiveFilesStorageInfo ready for production use (#9868)	3 years ago
db_bench_tool.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
db_dump_tool.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
db_stress_tool.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
env.h	typo fix: delete duplicate comment word (#9249)	3 years ago
env_encryption.h	Some API clarifications (#9080)	3 years ago
experimental.h	Add manifest fix-up utility for file temperatures (#9683)	3 years ago
file_checksum.h	Mark destructors as override (#9404)	3 years ago
file_system.h	Address comments for PR #9988 and #9996 (#10020)	3 years ago
filter_policy.h	Fix a major performance bug in 7.0 re: filter compatibility (#9736)	3 years ago
flush_block_policy.h	Some API clarifications (#9080)	3 years ago
functor_wrapper.h	Fix and detect headers with missing dependencies (#8893)	3 years ago
io_status.h	Combine data members of IOStatus with Status (#9549)	3 years ago
iostats_context.h	Remove ROCKSDB_SUPPORT_THREAD_LOCAL define because it's a part of C++11 (#10015)	3 years ago
iterator.h	Fix a few documentation errors including in public APIs (#9789)	3 years ago
ldb_tool.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
listener.h	Add temperature information to the event listener callbacks (#9591)	3 years ago
memory_allocator.h	Make MemoryAllocator into a Customizable class (#8980)	3 years ago
memtablerep.h	Added GetFactoryCount/Names/Types to ObjectRegistry (#9358)	3 years ago
merge_operator.h	Fix compile warnings (#9199)	3 years ago
metadata.h	Fix FileStorageInfo fields from GetLiveFilesMetaData (#9769)	3 years ago
options.h	Track SST unique id in MANIFEST and verify (#9990)	3 years ago
perf_context.h	Seek parallelization (#9994)	3 years ago
perf_level.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
persistent_cache.h	Check for and disallow shared key space in block caches (#9172)	3 years ago
rate_limiter.h	Remove deprecated option new_table_reader_for_compaction_inputs (#9443)	3 years ago
rocksdb_namespace.h	Fix and detect headers with missing dependencies (#8893)	3 years ago
secondary_cache.h	Prevent double caching in the compressed secondary cache (#9747)	3 years ago
slice.h	Require C++17 (#9481)	3 years ago
slice_transform.h	Some better API and other comments (#9533)	3 years ago
snapshot.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
sst_dump_tool.h	Add --version and --help to ldb and sst_dump (#6951)	5 years ago
sst_file_manager.h	Some API clarifications (#9080)	3 years ago
sst_file_reader.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
sst_file_writer.h	Support timestamps in SstFileWriter (#8899)	3 years ago
sst_partitioner.h	Mark destructors as override (#9404)	3 years ago
statistics.h	Multi file concurrency in MultiGet using coroutines and async IO (#9968)	3 years ago
stats_history.h	More refactoring ahead of footer & meta changes (#9240)	3 years ago
status.h	Combine data members of IOStatus with Status (#9549)	3 years ago
system_clock.h	Fix compile warnings (#9199)	3 years ago
table.h	Rewrite memory-charging feature's option API (#9926)	3 years ago
table_properties.h	Account memory of big memory users in BlockBasedTable in global memory limit (#9748)	3 years ago
thread_status.h	Remove ROCKSDB_SUPPORT_THREAD_LOCAL define because it's a part of C++11 (#10015)	3 years ago
threadpool.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
trace_reader_writer.h	Update comments, fix typos. (#8721)	3 years ago
trace_record.h	Add `IteratorTraceExecutionResult` for iterator related trace records. (#8687)	3 years ago
trace_record_result.h	Add `IteratorTraceExecutionResult` for iterator related trace records. (#8687)	3 years ago
transaction_log.h	Replace most typedef with using= (#8751)	3 years ago
types.h	Expose blob file information through the EventListener interface (#8675)	3 years ago
unique_id.h	Adjust public APIs to prefer 128-bit SST unique ID (#10009)	3 years ago
universal_compaction.h	Incremental Space Amp Compactions in Universal Style (#8655)	3 years ago
version.h	Update main version.h to NEXT release (7.3) (#9852)	3 years ago
wal_filter.h	Fix compile warnings (#9199)	3 years ago
write_batch.h	Fix various spelling errors still found in code (#9653)	3 years ago
write_batch_base.h	Revise APIs related to user-defined timestamp (#8946)	3 years ago
write_buffer_manager.h	Account memory of big memory users in BlockBasedTable in global memory limit (#9748)	3 years ago