rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

History

Hui Xiao bab5f9a6f2 Add new stat rocksdb.table.open.prefetch.tail.read.bytes, rocksdb.table.open.prefetch.tail.{miss\|hit} (#11265 ) Summary: Context/Summary: We are adding new stats to measure behavior of prefetched tail size and look up into this buffer The stat collection is done in FilePrefetchBuffer but only for prefetched tail buffer during table open for now using FilePrefetchBuffer enum. It's cleaner than the alternative of implementing in upper-level call places of FilePrefetchBuffer for table open. It also has the benefit of extensible to other types of FilePrefetchBuffer if needed. See db bench for perf regression concern. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11265 Test Plan: - Piggyback on existing test - rocksdb.table.open.prefetch.tail.miss is harder to UT so I manually set prefetch tail read bytes to be small and run db bench. ``` ./db_bench -db=/tmp/testdb -statistics=true -benchmarks="fillseq" -key_size=32 -value_size=512 -num=5000 -write_buffer_size=655 -target_file_size_base=655 -disable_auto_compactions=false -compression_type=none -bloom_bits=3 -use_direct_reads=true ``` ``` rocksdb.table.open.prefetch.tail.read.bytes P50 : 4096.000000 P95 : 4096.000000 P99 : 4096.000000 P100 : 4096.000000 COUNT : 225 SUM : 921600 rocksdb.table.open.prefetch.tail.miss COUNT : 91 rocksdb.table.open.prefetch.tail.hit COUNT : 1034 ``` - No perf regression observed in db_bench SETUP command: create same db with ~900 files for pre-change/post-change. ``` ./db_bench -db=/tmp/testdb -benchmarks="fillseq" -key_size=32 -value_size=512 -num=500000 -write_buffer_size=655360 -disable_auto_compactions=true -target_file_size_base=16777216 -compression_type=none ``` TEST command 60 runs or til convergence: as suggested by anand1976 and akankshamahajan15, vary `seek_nexts` and `async_io` in testing. ``` ./db_bench -use_existing_db=true -db=/tmp/testdb -statistics=false -cache_size=0 -cache_index_and_filter_blocks=false -benchmarks=seekrandom[-X60] -num=50000 -seek_nexts={10, 500, 1000} -async_io={0\|1} -use_direct_reads=true ``` async io = 0, direct io read = true \| seek_nexts = 10, 30 runs \| seek_nexts = 500, 12 runs \| seek_nexts = 1000, 6 runs -- \| -- \| -- \| -- pre-post change \| 4776 (± 28) ops/sec; 24.8 (± 0.1) MB/sec \| 288 (± 1) ops/sec; 74.8 (± 0.4) MB/sec \| 145 (± 4) ops/sec; 75.6 (± 2.2) MB/sec post-change \| 4790 (± 32) ops/sec; 24.9 (± 0.2) MB/sec \| 288 (± 3) ops/sec; 74.7 (± 0.8) MB/sec \| 143 (± 3) ops/sec; 74.5 (± 1.6) MB/sec async io = 1, direct io read = true \| seek_nexts = 10, 54 runs \| seek_nexts = 500, 6 runs \| seek_nexts = 1000, 4 runs -- \| -- \| -- \| -- pre-post change \| 3350 (± 36) ops/sec; 17.4 (± 0.2) MB/sec \| 264 (± 0) ops/sec; 68.7 (± 0.2) MB/sec \| 138 (± 1) ops/sec; 71.8 (± 1.0) MB/sec post-change \| 3358 (± 27) ops/sec; 17.4 (± 0.1) MB/sec \| 263 (± 2) ops/sec; 68.3 (± 0.8) MB/sec \| 139 (± 1) ops/sec; 72.6 (± 0.6) MB/sec Reviewed By: ajkr Differential Revision: D43781467 Pulled By: hx235 fbshipit-source-id: a706a18472a8edb2b952bac3af40eec803537f2a		2 years ago
..
binary_search_index_reader.cc	Set Read rate limiter priority dynamically and pass it to FS (#9996 )	3 years ago
binary_search_index_reader.h	Extend Get/MultiGet deadline support to table open (#6982 )	5 years ago
block.cc	Add API for writing wide-column entities (#10242 )	3 years ago
block.h	Put Cache and CacheWrapper in new public header (#11192 )	2 years ago
block_based_table_builder.cc	Remove compressed block cache (#11117 )	2 years ago
block_based_table_builder.h	Major Cache refactoring, CPU efficiency improvement (#10975 )	2 years ago
block_based_table_factory.cc	Remove a couple deprecated convenience.h APIs (#11120 )	2 years ago
block_based_table_factory.h	Remove RocksDB LITE (#11147 )	2 years ago
block_based_table_iterator.cc	Provide support for direct_reads with async_io (#10197 )	3 years ago
block_based_table_iterator.h	Format files under table/ by clang-format (#10852 )	2 years ago
block_based_table_reader.cc	Add new stat rocksdb.table.open.prefetch.tail.read.bytes, rocksdb.table.open.prefetch.tail.{miss\|hit} (#11265 )	2 years ago
block_based_table_reader.h	Add new stat rocksdb.table.open.prefetch.tail.read.bytes, rocksdb.table.open.prefetch.tail.{miss\|hit} (#11265 )	2 years ago
block_based_table_reader_impl.h	Major Cache refactoring, CPU efficiency improvement (#10975 )	2 years ago
block_based_table_reader_sync_and_async.h	Remove compressed block cache (#11117 )	2 years ago
block_based_table_reader_test.cc	Add a new MultiGetEntity API (#11222 )	2 years ago
block_builder.cc	Format files under table/ by clang-format (#10852 )	2 years ago
block_builder.h	Format files under table/ by clang-format (#10852 )	2 years ago
block_cache.cc	Major Cache refactoring, CPU efficiency improvement (#10975 )	2 years ago
block_cache.h	Remove compressed block cache (#11117 )	2 years ago
block_prefetcher.cc	Fix stress test failure for async_io (#10660 )	2 years ago
block_prefetcher.h	Provide support for direct_reads with async_io (#10197 )	3 years ago
block_prefix_index.cc	Fix bug with kHashSearch and changing prefix_extractor with SetOptions (#10128 )	3 years ago
block_prefix_index.h	Fix bug with kHashSearch and changing prefix_extractor with SetOptions (#10128 )	3 years ago
block_test.cc	Print stack traces on frozen tests in CI (#10828 )	2 years ago
block_type.h	Remove deprecated block-based filter (#10184 )	3 years ago
cachable_entry.h	Put Cache and CacheWrapper in new public header (#11192 )	2 years ago
data_block_footer.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
data_block_footer.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
data_block_hash_index.cc	Format files under table/ by clang-format (#10852 )	2 years ago
data_block_hash_index.h	Fix build with gcc 13 by including <cstdint> (#11118 )	2 years ago
data_block_hash_index_test.cc	Format files under table/ by clang-format (#10852 )	2 years ago
filter_block.h	Format files under table/ by clang-format (#10852 )	2 years ago
filter_block_reader_common.cc	Major Cache refactoring, CPU efficiency improvement (#10975 )	2 years ago
filter_block_reader_common.h	Major Cache refactoring, CPU efficiency improvement (#10975 )	2 years ago
filter_policy.cc	Remove RocksDB LITE (#11147 )	2 years ago
filter_policy_internal.h	Remove deprecated block-based filter (#10184 )	3 years ago
flush_block_policy.cc	Remove FactoryFunc from LoadXXXObject (#11203 )	2 years ago
flush_block_policy.h	Make FlushBlockPolicyFactory into a Customizable class (#8432 )	4 years ago
full_filter_block.cc	Major Cache refactoring, CPU efficiency improvement (#10975 )	2 years ago
full_filter_block.h	Format files under table/ by clang-format (#10852 )	2 years ago
full_filter_block_test.cc	Format files under table/ by clang-format (#10852 )	2 years ago
hash_index_reader.cc	Fix bug with kHashSearch and changing prefix_extractor with SetOptions (#10128 )	3 years ago
hash_index_reader.h	Extend Get/MultiGet deadline support to table open (#6982 )	5 years ago
index_builder.cc	Make InternalKeyComparator not configurable (#10342 )	2 years ago
index_builder.h	Format files under table/ by clang-format (#10852 )	2 years ago
index_reader_common.cc	Major Cache refactoring, CPU efficiency improvement (#10975 )	2 years ago
index_reader_common.h	Format files under table/ by clang-format (#10852 )	2 years ago
mock_block_based_table.h	Remove deprecated block-based filter (#10184 )	3 years ago
parsed_full_filter_block.cc	Hide FilterBits{Builder,Reader} from public API (#9592 )	3 years ago
parsed_full_filter_block.h	Major Cache refactoring, CPU efficiency improvement (#10975 )	2 years ago
partitioned_filter_block.cc	Major Cache refactoring, CPU efficiency improvement (#10975 )	2 years ago
partitioned_filter_block.h	Major Cache refactoring, CPU efficiency improvement (#10975 )	2 years ago
partitioned_filter_block_test.cc	Major Cache refactoring, CPU efficiency improvement (#10975 )	2 years ago
partitioned_index_iterator.cc	Provide support for direct_reads with async_io (#10197 )	3 years ago
partitioned_index_iterator.h	Format files under table/ by clang-format (#10852 )	2 years ago
partitioned_index_reader.cc	Use CacheDependencies() at start of ApproximateKeyAnchors() (#11230 )	2 years ago
partitioned_index_reader.h	Meta-internal folly integration with F14FastMap (#9546 )	3 years ago
reader_common.cc	Remove own ToString() (#9955 )	3 years ago
reader_common.h	Put Cache and CacheWrapper in new public header (#11192 )	2 years ago
uncompression_dict_reader.cc	Major Cache refactoring, CPU efficiency improvement (#10975 )	2 years ago
uncompression_dict_reader.h	Format files under table/ by clang-format (#10852 )	2 years ago