fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Hui Xiao 0f6d363db9 Fix higher read qps during db open caused by pr 11406 (#11516) ... Summary: **Context:** [PR11406](https://github.com/facebook/rocksdb/pull/11406/) caused more frequent read during db open reading files with no `tail_size` in the manifest as part of the upgrade to 11406. This is due to that PR introduced - [smaller](https://github.com/facebook/rocksdb/pull/11406/files#diff-57ed8c49db2bdd4db7618646a177397674bbf25beacacecb104070071d30129fR833) prefetch tail buffer size compared to pre-11406 for small files (< 52 MB) when `tail_prefetch_stats` infers tail size to be 0 (usually happens when the stats does not have much historical data to infer early on) - more read (up to # of partitioned filter/index) when such small prefetch tail buffer does not contain all the partitioned filter/index needed in CacheDependencies() since the [fallback logic](https://github.com/facebook/rocksdb/pull/11406/files#diff-d98f1a83de24412ad7f3527725dae7e28851c7222622c3cdb832d3cdf24bbf9fR165-R179) that prefetches all partitions at once will be [skipped](url) when such a small prefetch tail buffer is passed in **Summary:** - Revert the fallback prefetch buffer size change to preserve existing behavior fully during upgrading in `BlockBasedTable::PrefetchTail()` - Use passed-in prefetch tail buffer in `CacheDependencies()` only if it has a smaller offset than the the offset of first partition filter/index, that is, at least as good as the existing prefetching behavior Pull Request resolved: https://github.com/facebook/rocksdb/pull/11516 Test Plan: - db bench Create db with small files prior to PR 11406 ``` ./db_bench -db=/tmp/testdb/ --partition_index_and_filters=1 --statistics=1 -benchmarks=fillseq -key_size=3200 -value_size=5 -num=1000000 -write_buffer_size=6550000 -disable_auto_compactions=true -compression_type=zstd` ``` Read db to see if post-pr has lower read qps (i.e, rocksdb.file.read.db.open.micros count) during db open. ``` ./db_bench -use_direct_reads=1 --file_opening_threads=1 --threads=1 --use_existing_db=1 --seed=1682546046158958 --partition_index_and_filters=1 --statistics=1 --db=/tmp/testdb/ --benchmarks=readrandom --key_size=3200 --value_size=5 --num=100 --disable_auto_compactions=true --compression_type=zstd ``` Pre-PR: ``` rocksdb.file.read.db.open.micros P50 : 3.399023 P95 : 5.924468 P99 : 12.408333 P100 : 29.000000 COUNT : 611 SUM : 2539 ``` Post-PR: ``` rocksdb.file.read.db.open.micros P50 : 593.736842 P95 : 861.605263 P99 : 1212.868421 P100 : 2663.000000 COUNT : 585 SUM : 345349 ``` _Note: To control the starting offset of the prefetch tail buffer easier, I manually override the following to eliminate the effect of alignment_ ``` class PosixRandomAccessFile : public FSRandomAccessFile { virtual size_t GetRequiredBufferAlignment() const override { - return logical_sector_size_; + return 1; } ``` - CI Reviewed By: pdillinger Differential Revision: D46472566 Pulled By: hx235 fbshipit-source-id: 2fe14ac8d489d15b0e08e6f8fe4f46d5f110978e		2 years ago
..
delete_scheduler.cc	Remove RocksDB LITE (#11147)	2 years ago
delete_scheduler.h	Remove RocksDB LITE (#11147)	2 years ago
delete_scheduler_test.cc	Remove RocksDB LITE (#11147)	2 years ago
file_prefetch_buffer.cc	Change internal headers with duplicate names (#11408)	2 years ago
file_prefetch_buffer.h	Fix higher read qps during db open caused by pr 11406 (#11516)	2 years ago
file_util.cc	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288)	2 years ago
file_util.h	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288)	2 years ago
filename.cc	Run Clang format on file folder (#10860)	2 years ago
filename.h	Run Clang format on file folder (#10860)	2 years ago
line_file_reader.cc	Support read rate-limiting in SequentialFileReader (#9973)	3 years ago
line_file_reader.h	Support read rate-limiting in SequentialFileReader (#9973)	3 years ago
prefetch_test.cc	Record and use the tail size to prefetch table tail (#11406)	2 years ago
random_access_file_reader.cc	Add `rocksdb.file.read.db.open.micros` (#11455)	2 years ago
random_access_file_reader.h	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288)	2 years ago
random_access_file_reader_test.cc	Remove RocksDB LITE (#11147)	2 years ago
read_write_util.cc	Run Clang format on file folder (#10860)	2 years ago
read_write_util.h	Run Clang format on file folder (#10860)	2 years ago
readahead_file_info.h	Reuse internal auto readhead_size at each Level (expect L0) for Iterations (#9056)	3 years ago
readahead_raf.cc	Change internal headers with duplicate names (#11408)	2 years ago
readahead_raf.h	Make StringEnv, StringSink, StringSource use FS classes (#7786)	4 years ago
sequence_file_reader.cc	Change internal headers with duplicate names (#11408)	2 years ago
sequence_file_reader.h	Remove RocksDB LITE (#11147)	2 years ago
sst_file_manager_impl.cc	Remove RocksDB LITE (#11147)	2 years ago
sst_file_manager_impl.h	Remove RocksDB LITE (#11147)	2 years ago
writable_file_writer.cc	Change internal headers with duplicate names (#11408)	2 years ago
writable_file_writer.h	Remove RocksDB LITE (#11147)	2 years ago