You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Hui Xiao
bab5f9a6f2
Add new stat rocksdb.table.open.prefetch.tail.read.bytes, rocksdb.table.open.prefetch.tail.{miss|hit} (#11265)
Summary:
**Context/Summary:**
We are adding new stats to measure behavior of prefetched tail size and look up into this buffer
The stat collection is done in FilePrefetchBuffer but only for prefetched tail buffer during table open for now using FilePrefetchBuffer enum. It's cleaner than the alternative of implementing in upper-level call places of FilePrefetchBuffer for table open. It also has the benefit of extensible to other types of FilePrefetchBuffer if needed. See db bench for perf regression concern.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/11265
Test Plan:
**- Piggyback on existing test**
**- rocksdb.table.open.prefetch.tail.miss is harder to UT so I manually set prefetch tail read bytes to be small and run db bench.**
```
./db_bench -db=/tmp/testdb -statistics=true -benchmarks="fillseq" -key_size=32 -value_size=512 -num=5000 -write_buffer_size=655 -target_file_size_base=655 -disable_auto_compactions=false -compression_type=none -bloom_bits=3 -use_direct_reads=true
```
```
rocksdb.table.open.prefetch.tail.read.bytes P50 : 4096.000000 P95 : 4096.000000 P99 : 4096.000000 P100 : 4096.000000 COUNT : 225 SUM : 921600
rocksdb.table.open.prefetch.tail.miss COUNT : 91
rocksdb.table.open.prefetch.tail.hit COUNT : 1034
```
**- No perf regression observed in db_bench**
SETUP command: create same db with ~900 files for pre-change/post-change.
```
./db_bench -db=/tmp/testdb -benchmarks="fillseq" -key_size=32 -value_size=512 -num=500000 -write_buffer_size=655360 -disable_auto_compactions=true -target_file_size_base=16777216 -compression_type=none
```
TEST command 60 runs or til convergence: as suggested by anand1976 and akankshamahajan15, vary `seek_nexts` and `async_io` in testing.
```
./db_bench -use_existing_db=true -db=/tmp/testdb -statistics=false -cache_size=0 -cache_index_and_filter_blocks=false -benchmarks=seekrandom[-X60] -num=50000 -seek_nexts={10, 500, 1000} -async_io={0|1} -use_direct_reads=true
```
async io = 0, direct io read = true
| seek_nexts = 10, 30 runs | seek_nexts = 500, 12 runs | seek_nexts = 1000, 6 runs
-- | -- | -- | --
pre-post change | 4776 (± 28) ops/sec; 24.8 (± 0.1) MB/sec | 288 (± 1) ops/sec; 74.8 (± 0.4) MB/sec | 145 (± 4) ops/sec; 75.6 (± 2.2) MB/sec
post-change | 4790 (± 32) ops/sec; 24.9 (± 0.2) MB/sec | 288 (± 3) ops/sec; 74.7 (± 0.8) MB/sec | 143 (± 3) ops/sec; 74.5 (± 1.6) MB/sec
async io = 1, direct io read = true
| seek_nexts = 10, 54 runs | seek_nexts = 500, 6 runs | seek_nexts = 1000, 4 runs
-- | -- | -- | --
pre-post change | 3350 (± 36) ops/sec; 17.4 (± 0.2) MB/sec | 264 (± 0) ops/sec; 68.7 (± 0.2) MB/sec | 138 (± 1) ops/sec; 71.8 (± 1.0) MB/sec
post-change | 3358 (± 27) ops/sec; 17.4 (± 0.1) MB/sec | 263 (± 2) ops/sec; 68.3 (± 0.8) MB/sec | 139 (± 1) ops/sec; 72.6 (± 0.6) MB/sec
Reviewed By: ajkr
Differential Revision: D43781467
Pulled By: hx235
fbshipit-source-id: a706a18472a8edb2b952bac3af40eec803537f2a
|
2 years ago |
| .. |
|
adaptive
|
Remove RocksDB LITE (#11147)
|
2 years ago |
|
block_based
|
Add new stat rocksdb.table.open.prefetch.tail.read.bytes, rocksdb.table.open.prefetch.tail.{miss|hit} (#11265)
|
2 years ago |
|
cuckoo
|
Remove RocksDB LITE (#11147)
|
2 years ago |
|
plain
|
Remove a couple deprecated convenience.h APIs (#11120)
|
2 years ago |
|
block_fetcher.cc
|
Format files under table/ by clang-format (#10852)
|
2 years ago |
|
block_fetcher.h
|
Format files under table/ by clang-format (#10852)
|
2 years ago |
|
block_fetcher_test.cc
|
Remove RocksDB LITE (#11147)
|
2 years ago |
|
cleanable_test.cc
|
Eliminate unnecessary (slow) block cache Ref()ing in MultiGet (#9899)
|
3 years ago |
|
compaction_merging_iterator.cc
|
Refactor AddRangeDels() + consider range tombstone during compaction file cutting (#11113)
|
2 years ago |
|
compaction_merging_iterator.h
|
Refactor AddRangeDels() + consider range tombstone during compaction file cutting (#11113)
|
2 years ago |
|
format.cc
|
Remove RocksDB LITE (#11147)
|
2 years ago |
|
format.h
|
Major Cache refactoring, CPU efficiency improvement (#10975)
|
2 years ago |
|
get_context.cc
|
Rename a recently added PerfContext counter (#11294)
|
2 years ago |
|
get_context.h
|
Merge operator failed subcode (#11231)
|
2 years ago |
|
internal_iterator.h
|
remove unused InternalIteratorBase::is_mutable_ (#11104)
|
2 years ago |
|
iter_heap.h
|
Format files under table/ by clang-format (#10852)
|
2 years ago |
|
iterator.cc
|
Format files under table/ by clang-format (#10852)
|
2 years ago |
|
iterator_wrapper.h
|
Format files under table/ by clang-format (#10852)
|
2 years ago |
|
merger_test.cc
|
Print stack traces on frozen tests in CI (#10828)
|
2 years ago |
|
merging_iterator.cc
|
Improve documentation for MergingIterator (#11161)
|
2 years ago |
|
merging_iterator.h
|
Improve documentation for MergingIterator (#11161)
|
2 years ago |
|
meta_blocks.cc
|
Improve error messages for SST footer and size errors (#11009)
|
2 years ago |
|
meta_blocks.h
|
Refactor to avoid confusing "raw block" (#10408)
|
2 years ago |
|
mock_table.cc
|
Align compaction output file boundaries to the next level ones (#10655)
|
2 years ago |
|
mock_table.h
|
Align compaction output file boundaries to the next level ones (#10655)
|
2 years ago |
|
multiget_context.h
|
Add a new MultiGetEntity API (#11222)
|
2 years ago |
|
persistent_cache_helper.cc
|
Format files under table/ by clang-format (#10852)
|
2 years ago |
|
persistent_cache_helper.h
|
Refactor to avoid confusing "raw block" (#10408)
|
2 years ago |
|
persistent_cache_options.h
|
Use STATIC_AVOID_DESTRUCTION for static objects with non-trivial destructors (#9958)
|
3 years ago |
|
scoped_arena_iterator.h
|
Format files under table/ by clang-format (#10852)
|
2 years ago |
|
sst_file_dumper.cc
|
Remove RocksDB LITE (#11147)
|
2 years ago |
|
sst_file_dumper.h
|
Remove RocksDB LITE (#11147)
|
2 years ago |
|
sst_file_reader.cc
|
Remove RocksDB LITE (#11147)
|
2 years ago |
|
sst_file_reader_test.cc
|
Remove RocksDB LITE (#11147)
|
2 years ago |
|
sst_file_writer.cc
|
Remove RocksDB LITE (#11147)
|
2 years ago |
|
sst_file_writer_collectors.h
|
Refactor to avoid confusing "raw block" (#10408)
|
2 years ago |
|
table_builder.h
|
Always verify SST unique IDs on SST file open (#10532)
|
2 years ago |
|
table_factory.cc
|
Remove FactoryFunc from LoadXXXObject (#11203)
|
2 years ago |
|
table_properties.cc
|
Format files under table/ by clang-format (#10852)
|
2 years ago |
|
table_properties_internal.h
|
Improve / clean up meta block code & integrity (#9163)
|
3 years ago |
|
table_reader.h
|
Format files under table/ by clang-format (#10852)
|
2 years ago |
|
table_reader_bench.cc
|
Remove RocksDB LITE (#11147)
|
2 years ago |
|
table_test.cc
|
Tests verifying non-zero checksums of zero bytes (#11260)
|
2 years ago |
|
two_level_iterator.cc
|
Format files under table/ by clang-format (#10852)
|
2 years ago |
|
two_level_iterator.h
|
Format files under table/ by clang-format (#10852)
|
2 years ago |
|
unique_id.cc
|
Derive cache keys from SST unique IDs (#10394)
|
2 years ago |
|
unique_id_impl.h
|
Derive cache keys from SST unique IDs (#10394)
|
2 years ago |
