You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Hui Xiao
bab5f9a6f2
Add new stat rocksdb.table.open.prefetch.tail.read.bytes, rocksdb.table.open.prefetch.tail.{miss|hit} (#11265)
Summary:
**Context/Summary:**
We are adding new stats to measure behavior of prefetched tail size and look up into this buffer
The stat collection is done in FilePrefetchBuffer but only for prefetched tail buffer during table open for now using FilePrefetchBuffer enum. It's cleaner than the alternative of implementing in upper-level call places of FilePrefetchBuffer for table open. It also has the benefit of extensible to other types of FilePrefetchBuffer if needed. See db bench for perf regression concern.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/11265
Test Plan:
**- Piggyback on existing test**
**- rocksdb.table.open.prefetch.tail.miss is harder to UT so I manually set prefetch tail read bytes to be small and run db bench.**
```
./db_bench -db=/tmp/testdb -statistics=true -benchmarks="fillseq" -key_size=32 -value_size=512 -num=5000 -write_buffer_size=655 -target_file_size_base=655 -disable_auto_compactions=false -compression_type=none -bloom_bits=3 -use_direct_reads=true
```
```
rocksdb.table.open.prefetch.tail.read.bytes P50 : 4096.000000 P95 : 4096.000000 P99 : 4096.000000 P100 : 4096.000000 COUNT : 225 SUM : 921600
rocksdb.table.open.prefetch.tail.miss COUNT : 91
rocksdb.table.open.prefetch.tail.hit COUNT : 1034
```
**- No perf regression observed in db_bench**
SETUP command: create same db with ~900 files for pre-change/post-change.
```
./db_bench -db=/tmp/testdb -benchmarks="fillseq" -key_size=32 -value_size=512 -num=500000 -write_buffer_size=655360 -disable_auto_compactions=true -target_file_size_base=16777216 -compression_type=none
```
TEST command 60 runs or til convergence: as suggested by anand1976 and akankshamahajan15, vary `seek_nexts` and `async_io` in testing.
```
./db_bench -use_existing_db=true -db=/tmp/testdb -statistics=false -cache_size=0 -cache_index_and_filter_blocks=false -benchmarks=seekrandom[-X60] -num=50000 -seek_nexts={10, 500, 1000} -async_io={0|1} -use_direct_reads=true
```
async io = 0, direct io read = true
| seek_nexts = 10, 30 runs | seek_nexts = 500, 12 runs | seek_nexts = 1000, 6 runs
-- | -- | -- | --
pre-post change | 4776 (± 28) ops/sec; 24.8 (± 0.1) MB/sec | 288 (± 1) ops/sec; 74.8 (± 0.4) MB/sec | 145 (± 4) ops/sec; 75.6 (± 2.2) MB/sec
post-change | 4790 (± 32) ops/sec; 24.9 (± 0.2) MB/sec | 288 (± 3) ops/sec; 74.7 (± 0.8) MB/sec | 143 (± 3) ops/sec; 74.5 (± 1.6) MB/sec
async io = 1, direct io read = true
| seek_nexts = 10, 54 runs | seek_nexts = 500, 6 runs | seek_nexts = 1000, 4 runs
-- | -- | -- | --
pre-post change | 3350 (± 36) ops/sec; 17.4 (± 0.2) MB/sec | 264 (± 0) ops/sec; 68.7 (± 0.2) MB/sec | 138 (± 1) ops/sec; 71.8 (± 1.0) MB/sec
post-change | 3358 (± 27) ops/sec; 17.4 (± 0.1) MB/sec | 263 (± 2) ops/sec; 68.3 (± 0.8) MB/sec | 139 (± 1) ops/sec; 72.6 (± 0.6) MB/sec
Reviewed By: ajkr
Differential Revision: D43781467
Pulled By: hx235
fbshipit-source-id: a706a18472a8edb2b952bac3af40eec803537f2a
|
2 years ago |
| .. |
|
delete_scheduler.cc
|
Remove RocksDB LITE (#11147)
|
2 years ago |
|
delete_scheduler.h
|
Remove RocksDB LITE (#11147)
|
2 years ago |
|
delete_scheduler_test.cc
|
Remove RocksDB LITE (#11147)
|
2 years ago |
|
file_prefetch_buffer.cc
|
Add new stat rocksdb.table.open.prefetch.tail.read.bytes, rocksdb.table.open.prefetch.tail.{miss|hit} (#11265)
|
2 years ago |
|
file_prefetch_buffer.h
|
Add new stat rocksdb.table.open.prefetch.tail.read.bytes, rocksdb.table.open.prefetch.tail.{miss|hit} (#11265)
|
2 years ago |
|
file_util.cc
|
Remove RocksDB LITE (#11147)
|
2 years ago |
|
file_util.h
|
Set Read rate limiter priority dynamically and pass it to FS (#9996)
|
3 years ago |
|
filename.cc
|
Run Clang format on file folder (#10860)
|
2 years ago |
|
filename.h
|
Run Clang format on file folder (#10860)
|
2 years ago |
|
line_file_reader.cc
|
Support read rate-limiting in SequentialFileReader (#9973)
|
3 years ago |
|
line_file_reader.h
|
Support read rate-limiting in SequentialFileReader (#9973)
|
3 years ago |
|
prefetch_test.cc
|
Add new stat rocksdb.table.open.prefetch.tail.read.bytes, rocksdb.table.open.prefetch.tail.{miss|hit} (#11265)
|
2 years ago |
|
random_access_file_reader.cc
|
Remove RocksDB LITE (#11147)
|
2 years ago |
|
random_access_file_reader.h
|
Remove RocksDB LITE (#11147)
|
2 years ago |
|
random_access_file_reader_test.cc
|
Remove RocksDB LITE (#11147)
|
2 years ago |
|
read_write_util.cc
|
Run Clang format on file folder (#10860)
|
2 years ago |
|
read_write_util.h
|
Run Clang format on file folder (#10860)
|
2 years ago |
|
readahead_file_info.h
|
Reuse internal auto readhead_size at each Level (expect L0) for Iterations (#9056)
|
3 years ago |
|
readahead_raf.cc
|
Make StringEnv, StringSink, StringSource use FS classes (#7786)
|
4 years ago |
|
readahead_raf.h
|
Make StringEnv, StringSink, StringSource use FS classes (#7786)
|
4 years ago |
|
sequence_file_reader.cc
|
Remove RocksDB LITE (#11147)
|
2 years ago |
|
sequence_file_reader.h
|
Remove RocksDB LITE (#11147)
|
2 years ago |
|
sst_file_manager_impl.cc
|
Remove RocksDB LITE (#11147)
|
2 years ago |
|
sst_file_manager_impl.h
|
Remove RocksDB LITE (#11147)
|
2 years ago |
|
writable_file_writer.cc
|
Remove RocksDB LITE (#11147)
|
2 years ago |
|
writable_file_writer.h
|
Remove RocksDB LITE (#11147)
|
2 years ago |
