You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Hui Xiao
bab5f9a6f2
Add new stat rocksdb.table.open.prefetch.tail.read.bytes, rocksdb.table.open.prefetch.tail.{miss|hit} (#11265)
Summary:
**Context/Summary:**
We are adding new stats to measure behavior of prefetched tail size and look up into this buffer
The stat collection is done in FilePrefetchBuffer but only for prefetched tail buffer during table open for now using FilePrefetchBuffer enum. It's cleaner than the alternative of implementing in upper-level call places of FilePrefetchBuffer for table open. It also has the benefit of extensible to other types of FilePrefetchBuffer if needed. See db bench for perf regression concern.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/11265
Test Plan:
**- Piggyback on existing test**
**- rocksdb.table.open.prefetch.tail.miss is harder to UT so I manually set prefetch tail read bytes to be small and run db bench.**
```
./db_bench -db=/tmp/testdb -statistics=true -benchmarks="fillseq" -key_size=32 -value_size=512 -num=5000 -write_buffer_size=655 -target_file_size_base=655 -disable_auto_compactions=false -compression_type=none -bloom_bits=3 -use_direct_reads=true
```
```
rocksdb.table.open.prefetch.tail.read.bytes P50 : 4096.000000 P95 : 4096.000000 P99 : 4096.000000 P100 : 4096.000000 COUNT : 225 SUM : 921600
rocksdb.table.open.prefetch.tail.miss COUNT : 91
rocksdb.table.open.prefetch.tail.hit COUNT : 1034
```
**- No perf regression observed in db_bench**
SETUP command: create same db with ~900 files for pre-change/post-change.
```
./db_bench -db=/tmp/testdb -benchmarks="fillseq" -key_size=32 -value_size=512 -num=500000 -write_buffer_size=655360 -disable_auto_compactions=true -target_file_size_base=16777216 -compression_type=none
```
TEST command 60 runs or til convergence: as suggested by anand1976 and akankshamahajan15, vary `seek_nexts` and `async_io` in testing.
```
./db_bench -use_existing_db=true -db=/tmp/testdb -statistics=false -cache_size=0 -cache_index_and_filter_blocks=false -benchmarks=seekrandom[-X60] -num=50000 -seek_nexts={10, 500, 1000} -async_io={0|1} -use_direct_reads=true
```
async io = 0, direct io read = true
| seek_nexts = 10, 30 runs | seek_nexts = 500, 12 runs | seek_nexts = 1000, 6 runs
-- | -- | -- | --
pre-post change | 4776 (± 28) ops/sec; 24.8 (± 0.1) MB/sec | 288 (± 1) ops/sec; 74.8 (± 0.4) MB/sec | 145 (± 4) ops/sec; 75.6 (± 2.2) MB/sec
post-change | 4790 (± 32) ops/sec; 24.9 (± 0.2) MB/sec | 288 (± 3) ops/sec; 74.7 (± 0.8) MB/sec | 143 (± 3) ops/sec; 74.5 (± 1.6) MB/sec
async io = 1, direct io read = true
| seek_nexts = 10, 54 runs | seek_nexts = 500, 6 runs | seek_nexts = 1000, 4 runs
-- | -- | -- | --
pre-post change | 3350 (± 36) ops/sec; 17.4 (± 0.2) MB/sec | 264 (± 0) ops/sec; 68.7 (± 0.2) MB/sec | 138 (± 1) ops/sec; 71.8 (± 1.0) MB/sec
post-change | 3358 (± 27) ops/sec; 17.4 (± 0.1) MB/sec | 263 (± 2) ops/sec; 68.3 (± 0.8) MB/sec | 139 (± 1) ops/sec; 72.6 (± 0.6) MB/sec
Reviewed By: ajkr
Differential Revision: D43781467
Pulled By: hx235
fbshipit-source-id: a706a18472a8edb2b952bac3af40eec803537f2a
|
2 years ago |
| .. |
|
file_read_sample.h
|
Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)
|
5 years ago |
|
histogram.cc
|
clang format files under monitoring/ (#10857)
|
2 years ago |
|
histogram.h
|
clang format files under monitoring/ (#10857)
|
2 years ago |
|
histogram_test.cc
|
clang format files under monitoring/ (#10857)
|
2 years ago |
|
histogram_windowing.cc
|
clang format files under monitoring/ (#10857)
|
2 years ago |
|
histogram_windowing.h
|
clang format files under monitoring/ (#10857)
|
2 years ago |
|
in_memory_stats_history.cc
|
clang format files under monitoring/ (#10857)
|
2 years ago |
|
in_memory_stats_history.h
|
Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)
|
5 years ago |
|
instrumented_mutex.cc
|
Inject spurious wakeup and sleep before acquiring db mutex to expose race condition (#10291)
|
2 years ago |
|
instrumented_mutex.h
|
clang format files under monitoring/ (#10857)
|
2 years ago |
|
iostats_context.cc
|
clang format files under monitoring/ (#10857)
|
2 years ago |
|
iostats_context_imp.h
|
Fix build error with NIOSTATS_CONTEXT (#10506)
|
2 years ago |
|
iostats_context_test.cc
|
clang format files under monitoring/ (#10857)
|
2 years ago |
|
perf_context.cc
|
Rename a recently added PerfContext counter (#11294)
|
2 years ago |
|
perf_context_imp.h
|
Remove ROCKSDB_SUPPORT_THREAD_LOCAL define because it's a part of C++11 (#10015)
|
3 years ago |
|
perf_level.cc
|
clang format files under monitoring/ (#10857)
|
2 years ago |
|
perf_level_imp.h
|
clang format files under monitoring/ (#10857)
|
2 years ago |
|
perf_step_timer.h
|
clang format files under monitoring/ (#10857)
|
2 years ago |
|
persistent_stats_history.cc
|
clang format files under monitoring/ (#10857)
|
2 years ago |
|
persistent_stats_history.h
|
Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)
|
5 years ago |
|
statistics.cc
|
Add new stat rocksdb.table.open.prefetch.tail.read.bytes, rocksdb.table.open.prefetch.tail.{miss|hit} (#11265)
|
2 years ago |
|
statistics.h
|
clang format files under monitoring/ (#10857)
|
2 years ago |
|
statistics_test.cc
|
Remove RocksDB LITE (#11147)
|
2 years ago |
|
stats_history_test.cc
|
Remove RocksDB LITE (#11147)
|
2 years ago |
|
thread_status_impl.cc
|
Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)
|
5 years ago |
|
thread_status_updater.cc
|
Remove ROCKSDB_SUPPORT_THREAD_LOCAL define because it's a part of C++11 (#10015)
|
3 years ago |
|
thread_status_updater.h
|
clang format files under monitoring/ (#10857)
|
2 years ago |
|
thread_status_updater_debug.cc
|
Replace reinterpret_cast with static_cast_with_check (#7067)
|
5 years ago |
|
thread_status_util.cc
|
Remove ROCKSDB_SUPPORT_THREAD_LOCAL define because it's a part of C++11 (#10015)
|
3 years ago |
|
thread_status_util.h
|
clang format files under monitoring/ (#10857)
|
2 years ago |
|
thread_status_util_debug.cc
|
clang format files under monitoring/ (#10857)
|
2 years ago |
