fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Changyu Bi fd165c869d Add memtable per key-value checksum (#10281) ... Summary: Append per key-value checksum to internal key. These checksums are verified on read paths including Get, Iterator and during Flush. Get and Iterator will return `Corruption` status if there is a checksum verification failure. Flush will make DB become read-only upon memtable entry checksum verification failure. Pull Request resolved: https://github.com/facebook/rocksdb/pull/10281 Test Plan: - Added new unit test cases: `make check` - Benchmark on memtable insert ``` TEST_TMPDIR=/dev/shm/memtable_write ./db_bench -benchmarks=fillseq -disable_wal=true -max_write_buffer_number=100 -num=10000000 -min_write_buffer_number_to_merge=100 # avg over 10 runs Baseline: 1166936 ops/sec memtable 2 bytes kv checksum : 1.11674e+06 ops/sec (-4%) memtable 2 bytes kv checksum + write batch 8 bytes kv checksum: 1.08579e+06 ops/sec (-6.95%) write batch 8 bytes kv checksum: 1.17979e+06 ops/sec (+1.1%) ``` - Benchmark on only memtable read: ops/sec dropped 31% for `readseq` due to time spend on verifying checksum. ops/sec for `readrandom` dropped ~6.8%. ``` # Readseq sudo TEST_TMPDIR=/dev/shm/memtable_read ./db_bench -benchmarks=fillseq,readseq"[-X20]" -disable_wal=true -max_write_buffer_number=100 -num=10000000 -min_write_buffer_number_to_merge=100 readseq [AVG 20 runs] : 7432840 (± 212005) ops/sec; 822.3 (± 23.5) MB/sec readseq [MEDIAN 20 runs] : 7573878 ops/sec; 837.9 MB/sec With -memtable_protection_bytes_per_key=2: readseq [AVG 20 runs] : 5134607 (± 119596) ops/sec; 568.0 (± 13.2) MB/sec readseq [MEDIAN 20 runs] : 5232946 ops/sec; 578.9 MB/sec # Readrandom sudo TEST_TMPDIR=/dev/shm/memtable_read ./db_bench -benchmarks=fillrandom,readrandom"[-X10]" -disable_wal=true -max_write_buffer_number=100 -num=1000000 -min_write_buffer_number_to_merge=100 readrandom [AVG 10 runs] : 140236 (± 3938) ops/sec; 9.8 (± 0.3) MB/sec readrandom [MEDIAN 10 runs] : 140545 ops/sec; 9.8 MB/sec With -memtable_protection_bytes_per_key=2: readrandom [AVG 10 runs] : 130632 (± 2738) ops/sec; 9.1 (± 0.2) MB/sec readrandom [MEDIAN 10 runs] : 130341 ops/sec; 9.1 MB/sec ``` - Stress test: `python3 -u tools/db_crashtest.py whitebox --duration=1800` Reviewed By: ajkr Differential Revision: D37607896 Pulled By: cbi42 fbshipit-source-id: fdaefb475629d2471780d4a5f5bf81b44ee56113		2 years ago
..
CMakeLists.txt	Stress test for RocksDB transactions (#8936)	3 years ago
batched_ops_stress.cc	Add rate-limiting support to batched MultiGet() (#10159)	3 years ago
cf_consistency_stress.cc	Add rate-limiting support to batched MultiGet() (#10159)	3 years ago
db_stress.cc	Add (& fix) some simple source code checks (#8821)	3 years ago
db_stress_common.cc	Stop operating on DB in a stress test background thread (#10373)	2 years ago
db_stress_common.h	Add memtable per key-value checksum (#10281)	2 years ago
db_stress_compaction_filter.h	Enable compaction filter for db_stress with user-defined timestamp (#10259)	3 years ago
db_stress_driver.cc	Stop operating on DB in a stress test background thread (#10373)	2 years ago
db_stress_driver.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
db_stress_env_wrapper.h	Make the Env class Customizable (#9293)	3 years ago
db_stress_gflags.cc	Add memtable per key-value checksum (#10281)	2 years ago
db_stress_listener.cc	Adjust public APIs to prefer 128-bit SST unique ID (#10009)	3 years ago
db_stress_listener.h	Avoid usage of ReopenWritableFile in db_stress (#9649)	3 years ago
db_stress_shared_state.cc	Remove ROCKSDB_SUPPORT_THREAD_LOCAL define because it's a part of C++11 (#10015)	3 years ago
db_stress_shared_state.h	Expand stress test coverage for user-defined timestamp (#10280)	3 years ago
db_stress_stat.cc	Fix Statistics in db_stress (#9260)	3 years ago
db_stress_stat.h	Fix Statistics in db_stress (#9260)	3 years ago
db_stress_table_properties_collector.h	Fix and detect headers with missing dependencies (#8893)	3 years ago
db_stress_test_base.cc	Add memtable per key-value checksum (#10281)	2 years ago
db_stress_test_base.h	Stop operating on DB in a stress test background thread (#10373)	2 years ago
db_stress_tool.cc	Add WriteOptions::protection_bytes_per_key (#10037)	3 years ago
expected_state.cc	Remove own ToString() (#9955)	3 years ago
expected_state.h	db_stress verify with lost unsynced operations (#8966)	3 years ago
multi_ops_txns_stress.cc	Stop operating on DB in a stress test background thread (#10373)	2 years ago
multi_ops_txns_stress.h	Add coverage for timestamped snapshot to MultiOpsTxnsStressTest (#10325)	2 years ago
no_batched_ops_stress.cc	Expand stress test coverage for user-defined timestamp (#10280)	3 years ago