fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Hui Xiao 151242ce46 Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288) ... Summary: **Context:** The existing stat rocksdb.sst.read.micros does not reflect each of compaction and flush cases but aggregate them, which is not so helpful for us to understand IO read behavior of each of them. **Summary** - Update `StopWatch` and `RandomAccessFileReader` to record `rocksdb.sst.read.micros` and `rocksdb.file.{flush/compaction}.read.micros` - Fixed the default histogram in `RandomAccessFileReader` - New field `ReadOptions/IOOptions::io_activity`; Pass `ReadOptions` through paths under db open, flush and compaction to where we can prepare `IOOptions` and pass it to `RandomAccessFileReader` - Use `thread_status_util` for assertion in `DbStressFSWrapper` for continuous testing on we are passing correct `io_activity` under db open, flush and compaction Pull Request resolved: https://github.com/facebook/rocksdb/pull/11288 Test Plan: - **Stress test** - **Db bench 1: rocksdb.sst.read.micros COUNT ≈ sum of rocksdb.file.read.flush.micros's and rocksdb.file.read.compaction.micros's.** (without blob) - May not be exactly the same due to `HistogramStat::Add` only guarantees atomic not accuracy across threads. ``` ./db_bench -db=/dev/shm/testdb/ -statistics=true -benchmarks="fillseq" -key_size=32 -value_size=512 -num=50000 -write_buffer_size=655 -target_file_size_base=655 -disable_auto_compactions=false -compression_type=none -bloom_bits=3 (-use_plain_table=1 -prefix_size=10) ``` ``` // BlockBasedTable rocksdb.sst.read.micros P50 : 2.009374 P95 : 4.968548 P99 : 8.110362 P100 : 43.000000 COUNT : 40456 SUM : 114805 rocksdb.file.read.flush.micros P50 : 1.871841 P95 : 3.872407 P99 : 5.540541 P100 : 43.000000 COUNT : 2250 SUM : 6116 rocksdb.file.read.compaction.micros P50 : 2.023109 P95 : 5.029149 P99 : 8.196910 P100 : 26.000000 COUNT : 38206 SUM : 108689 // PlainTable Does not apply ``` - **Db bench 2: performance** **Read** SETUP: db with 900 files ``` ./db_bench -db=/dev/shm/testdb/ -benchmarks="fillseq" -key_size=32 -value_size=512 -num=50000 -write_buffer_size=655 -disable_auto_compactions=true -target_file_size_base=655 -compression_type=none ```run till convergence ``` ./db_bench -seed=1678564177044286 -use_existing_db=true -db=/dev/shm/testdb -benchmarks=readrandom[-X60] -statistics=true -num=1000000 -disable_auto_compactions=true -compression_type=none -bloom_bits=3 ``` Pre-change `readrandom [AVG 60 runs] : 21568 (± 248) ops/sec` Post-change (no regression, -0.3%) `readrandom [AVG 60 runs] : 21486 (± 236) ops/sec` **Compaction/Flush**run till convergence ``` ./db_bench -db=/dev/shm/testdb2/ -seed=1678564177044286 -benchmarks="fillseq[-X60]" -key_size=32 -value_size=512 -num=50000 -write_buffer_size=655 -disable_auto_compactions=false -target_file_size_base=655 -compression_type=none rocksdb.sst.read.micros COUNT : 33820 rocksdb.sst.read.flush.micros COUNT : 1800 rocksdb.sst.read.compaction.micros COUNT : 32020 ``` Pre-change `fillseq [AVG 46 runs] : 1391 (± 214) ops/sec; 0.7 (± 0.1) MB/sec` Post-change (no regression, ~-0.4%) `fillseq [AVG 46 runs] : 1385 (± 216) ops/sec; 0.7 (± 0.1) MB/sec` Reviewed By: ajkr Differential Revision: D44007011 Pulled By: hx235 fbshipit-source-id: a54c89e4846dfc9a135389edf3f3eedfea257132		2 years ago
..
CMakeLists.txt	Stress test for RocksDB transactions (#8936)	3 years ago
batched_ops_stress.cc	Extend the stress test coverage of MultiGetEntity (#11336)	2 years ago
cf_consistency_stress.cc	Extend the stress test coverage of MultiGetEntity (#11336)	2 years ago
db_stress.cc	Disable tiered storage + BlobDB stress test (#10699)	2 years ago
db_stress_common.cc	Increase the stress test coverage of GetEntity (#11303)	2 years ago
db_stress_common.h	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288)	2 years ago
db_stress_compaction_filter.h	Enable compaction filter for db_stress with user-defined timestamp (#10259)	2 years ago
db_stress_driver.cc	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288)	2 years ago
db_stress_driver.h	fix shared state used after free (#11059)	2 years ago
db_stress_env_wrapper.h	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288)	2 years ago
db_stress_gflags.cc	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288)	2 years ago
db_stress_listener.cc	Remove RocksDB LITE (#11147)	2 years ago
db_stress_listener.h	Remove RocksDB LITE (#11147)	2 years ago
db_stress_shared_state.cc	Remove ROCKSDB_SUPPORT_THREAD_LOCAL define because it's a part of C++11 (#10015)	3 years ago
db_stress_shared_state.h	Run clang format against files under tools/ and db_stress_tool/ (#10868)	2 years ago
db_stress_stat.cc	Fix Statistics in db_stress (#9260)	3 years ago
db_stress_stat.h	Fix Statistics in db_stress (#9260)	3 years ago
db_stress_table_properties_collector.h	Fix and detect headers with missing dependencies (#8893)	3 years ago
db_stress_test_base.cc	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288)	2 years ago
db_stress_test_base.h	Extend the stress test coverage of MultiGetEntity (#11336)	2 years ago
db_stress_tool.cc	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288)	2 years ago
expected_state.cc	Increase the stress test coverage of GetEntity (#11303)	2 years ago
expected_state.h	db_stress verify with lost unsynced operations (#8966)	3 years ago
multi_ops_txns_stress.cc	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288)	2 years ago
multi_ops_txns_stress.h	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288)	2 years ago
no_batched_ops_stress.cc	Extend the stress test coverage of MultiGetEntity (#11336)	2 years ago