fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Changyu Bi 62fc15f009 Block per key-value checksum (#11287) ... Summary: add option `block_protection_bytes_per_key` and implementation for block per key-value checksum. The main changes are 1. checksum construction and verification in block.cc/h 2. pass the option `block_protection_bytes_per_key` around (mainly for methods defined in table_cache.h) 3. unit tests/crash test updates Tests: * Added unit tests * Crash test: `python3 tools/db_crashtest.py blackbox --simple --block_protection_bytes_per_key=1 --write_buffer_size=1048576` Follow up (maybe as a separate PR): make sure corruption status returned from BlockIters are correctly handled. Performance: Turning on block per KV protection has a non-trivial negative impact on read performance and costs additional memory. For memory, each block includes additional 24 bytes for checksum-related states beside checksum itself. For CPU, I set up a DB of size ~1.2GB with 5M keys (32 bytes key and 200 bytes value) which compacts to ~5 SST files (target file size 256 MB) in L6 without compression. I tested readrandom performance with various block cache size (to mimic various cache hit rates): ``` SETUP make OPTIMIZE_LEVEL="-O3" USE_LTO=1 DEBUG_LEVEL=0 -j32 db_bench ./db_bench -benchmarks=fillseq,compact0,waitforcompaction,compact,waitforcompaction -write_buffer_size=33554432 -level_compaction_dynamic_level_bytes=true -max_background_jobs=8 -target_file_size_base=268435456 --num=5000000 --key_size=32 --value_size=200 --compression_type=none BENCHMARK ./db_bench --use_existing_db -benchmarks=readtocache,readrandom[-X10] --num=5000000 --key_size=32 --disable_auto_compactions --reads=1000000 --block_protection_bytes_per_key=[0|1] --cache_size=$CACHESIZE The readrandom ops/sec looks like the following: Block cache size: 2GB 1.2GB * 0.9 1.2GB * 0.8 1.2GB * 0.5 8MB Main 240805 223604 198176 161653 139040 PR prot_bytes=0 238691 226693 200127 161082 141153 PR prot_bytes=1 214983 193199 178532 137013 108211 prot_bytes=1 vs -10% -15% -10.8% -15% -23% prot_bytes=0 ``` The benchmark has a lot of variance, but there was a 5% to 25% regression in this benchmark with different cache hit rates. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11287 Reviewed By: ajkr Differential Revision: D43970708 Pulled By: cbi42 fbshipit-source-id: ef98d898b71779846fa74212b9ec9e08b7183940		2 years ago
..
CMakeLists.txt	Stress test for RocksDB transactions (#8936)	3 years ago
batched_ops_stress.cc	Extend the stress test coverage of MultiGetEntity (#11336)	2 years ago
cf_consistency_stress.cc	Extend the stress test coverage of MultiGetEntity (#11336)	2 years ago
db_stress.cc	Disable tiered storage + BlobDB stress test (#10699)	2 years ago
db_stress_common.cc	Increase the stress test coverage of GetEntity (#11303)	2 years ago
db_stress_common.h	Block per key-value checksum (#11287)	2 years ago
db_stress_compaction_filter.h	Enable compaction filter for db_stress with user-defined timestamp (#10259)	2 years ago
db_stress_driver.cc	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288)	2 years ago
db_stress_driver.h	fix shared state used after free (#11059)	2 years ago
db_stress_env_wrapper.h	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288)	2 years ago
db_stress_gflags.cc	Block per key-value checksum (#11287)	2 years ago
db_stress_listener.cc	Remove RocksDB LITE (#11147)	2 years ago
db_stress_listener.h	Remove RocksDB LITE (#11147)	2 years ago
db_stress_shared_state.cc	Remove ROCKSDB_SUPPORT_THREAD_LOCAL define because it's a part of C++11 (#10015)	3 years ago
db_stress_shared_state.h	Run clang format against files under tools/ and db_stress_tool/ (#10868)	2 years ago
db_stress_stat.cc	Fix Statistics in db_stress (#9260)	3 years ago
db_stress_stat.h	Fix Statistics in db_stress (#9260)	3 years ago
db_stress_table_properties_collector.h	Fix and detect headers with missing dependencies (#8893)	3 years ago
db_stress_test_base.cc	Block per key-value checksum (#11287)	2 years ago
db_stress_test_base.h	Extend the stress test coverage of MultiGetEntity (#11336)	2 years ago
db_stress_tool.cc	Add back io_uring stress test hack with DbStressFSWrapper for FS not supporting read async (#11404)	2 years ago
expected_state.cc	Increase the stress test coverage of GetEntity (#11303)	2 years ago
expected_state.h	db_stress verify with lost unsynced operations (#8966)	3 years ago
multi_ops_txns_stress.cc	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288)	2 years ago
multi_ops_txns_stress.h	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288)	2 years ago
no_batched_ops_stress.cc	Extend the stress test coverage of MultiGetEntity (#11336)	2 years ago