fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Hui Xiao 0f6d363db9 Fix higher read qps during db open caused by pr 11406 (#11516) ... Summary: **Context:** [PR11406](https://github.com/facebook/rocksdb/pull/11406/) caused more frequent read during db open reading files with no `tail_size` in the manifest as part of the upgrade to 11406. This is due to that PR introduced - [smaller](https://github.com/facebook/rocksdb/pull/11406/files#diff-57ed8c49db2bdd4db7618646a177397674bbf25beacacecb104070071d30129fR833) prefetch tail buffer size compared to pre-11406 for small files (< 52 MB) when `tail_prefetch_stats` infers tail size to be 0 (usually happens when the stats does not have much historical data to infer early on) - more read (up to # of partitioned filter/index) when such small prefetch tail buffer does not contain all the partitioned filter/index needed in CacheDependencies() since the [fallback logic](https://github.com/facebook/rocksdb/pull/11406/files#diff-d98f1a83de24412ad7f3527725dae7e28851c7222622c3cdb832d3cdf24bbf9fR165-R179) that prefetches all partitions at once will be [skipped](url) when such a small prefetch tail buffer is passed in **Summary:** - Revert the fallback prefetch buffer size change to preserve existing behavior fully during upgrading in `BlockBasedTable::PrefetchTail()` - Use passed-in prefetch tail buffer in `CacheDependencies()` only if it has a smaller offset than the the offset of first partition filter/index, that is, at least as good as the existing prefetching behavior Pull Request resolved: https://github.com/facebook/rocksdb/pull/11516 Test Plan: - db bench Create db with small files prior to PR 11406 ``` ./db_bench -db=/tmp/testdb/ --partition_index_and_filters=1 --statistics=1 -benchmarks=fillseq -key_size=3200 -value_size=5 -num=1000000 -write_buffer_size=6550000 -disable_auto_compactions=true -compression_type=zstd` ``` Read db to see if post-pr has lower read qps (i.e, rocksdb.file.read.db.open.micros count) during db open. ``` ./db_bench -use_direct_reads=1 --file_opening_threads=1 --threads=1 --use_existing_db=1 --seed=1682546046158958 --partition_index_and_filters=1 --statistics=1 --db=/tmp/testdb/ --benchmarks=readrandom --key_size=3200 --value_size=5 --num=100 --disable_auto_compactions=true --compression_type=zstd ``` Pre-PR: ``` rocksdb.file.read.db.open.micros P50 : 3.399023 P95 : 5.924468 P99 : 12.408333 P100 : 29.000000 COUNT : 611 SUM : 2539 ``` Post-PR: ``` rocksdb.file.read.db.open.micros P50 : 593.736842 P95 : 861.605263 P99 : 1212.868421 P100 : 2663.000000 COUNT : 585 SUM : 345349 ``` _Note: To control the starting offset of the prefetch tail buffer easier, I manually override the following to eliminate the effect of alignment_ ``` class PosixRandomAccessFile : public FSRandomAccessFile { virtual size_t GetRequiredBufferAlignment() const override { - return logical_sector_size_; + return 1; } ``` - CI Reviewed By: pdillinger Differential Revision: D46472566 Pulled By: hx235 fbshipit-source-id: 2fe14ac8d489d15b0e08e6f8fe4f46d5f110978e		2 years ago
..
adaptive	Remove RocksDB LITE (#11147)	2 years ago
block_based	Fix higher read qps during db open caused by pr 11406 (#11516)	2 years ago
cuckoo	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288)	2 years ago
plain	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288)	2 years ago
block_fetcher.cc	Change internal headers with duplicate names (#11408)	2 years ago
block_fetcher.h	Change internal headers with duplicate names (#11408)	2 years ago
block_fetcher_test.cc	Record and use the tail size to prefetch table tail (#11406)	2 years ago
cleanable_test.cc	Eliminate unnecessary (slow) block cache Ref()ing in MultiGet (#9899)	3 years ago
compaction_merging_iterator.cc	Refactor AddRangeDels() + consider range tombstone during compaction file cutting (#11113)	2 years ago
compaction_merging_iterator.h	Refactor AddRangeDels() + consider range tombstone during compaction file cutting (#11113)	2 years ago
format.cc	Change internal headers with duplicate names (#11408)	2 years ago
format.h	Change internal headers with duplicate names (#11408)	2 years ago
get_context.cc	Change internal headers with duplicate names (#11408)	2 years ago
get_context.h	Merge operator failed subcode (#11231)	2 years ago
internal_iterator.h	remove unused InternalIteratorBase::is_mutable_ (#11104)	2 years ago
iter_heap.h	Format files under table/ by clang-format (#10852)	2 years ago
iterator.cc	Format files under table/ by clang-format (#10852)	2 years ago
iterator_wrapper.h	Format files under table/ by clang-format (#10852)	2 years ago
merger_test.cc	Print stack traces on frozen tests in CI (#10828)	2 years ago
merging_iterator.cc	Improve documentation for MergingIterator (#11161)	2 years ago
merging_iterator.h	Improve documentation for MergingIterator (#11161)	2 years ago
meta_blocks.cc	Record and use the tail size to prefetch table tail (#11406)	2 years ago
meta_blocks.h	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288)	2 years ago
mock_table.cc	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288)	2 years ago
mock_table.h	Align compaction output file boundaries to the next level ones (#10655)	2 years ago
multiget_context.h	Add a new MultiGetEntity API (#11222)	2 years ago
persistent_cache_helper.cc	Format files under table/ by clang-format (#10852)	2 years ago
persistent_cache_helper.h	Change internal headers with duplicate names (#11408)	2 years ago
persistent_cache_options.h	Change internal headers with duplicate names (#11408)	2 years ago
scoped_arena_iterator.h	Format files under table/ by clang-format (#10852)	2 years ago
sst_file_dumper.cc	Block per key-value checksum (#11287)	2 years ago
sst_file_dumper.h	Remove RocksDB LITE (#11147)	2 years ago
sst_file_reader.cc	Block per key-value checksum (#11287)	2 years ago
sst_file_reader_test.cc	Remove RocksDB LITE (#11147)	2 years ago
sst_file_writer.cc	validate SstFileWriter range tombstones cover positive ranges (#11322)	2 years ago
sst_file_writer_collectors.h	Refactor to avoid confusing "raw block" (#10408)	2 years ago
table_builder.h	Record and use the tail size to prefetch table tail (#11406)	2 years ago
table_factory.cc	Remove FactoryFunc from LoadXXXObject (#11203)	2 years ago
table_properties.cc	Record and use the tail size to prefetch table tail (#11406)	2 years ago
table_properties_internal.h	Improve / clean up meta block code & integrity (#9163)	3 years ago
table_reader.h	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288)	2 years ago
table_reader_bench.cc	Block per key-value checksum (#11287)	2 years ago
table_test.cc	Change internal headers with duplicate names (#11408)	2 years ago
two_level_iterator.cc	Format files under table/ by clang-format (#10852)	2 years ago
two_level_iterator.h	Format files under table/ by clang-format (#10852)	2 years ago
unique_id.cc	Derive cache keys from SST unique IDs (#10394)	2 years ago
unique_id_impl.h	Derive cache keys from SST unique IDs (#10394)	2 years ago