fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Peter Dillinger 39f5846ec7 Much better stats for seeks and prefix filtering (#11460) ... Summary: We want to know more about opportunities for better range filters, and the effectiveness of our own range filters. Currently the stats are very limited, essentially logging just hits and misses against prefix filters for range scans in BLOOM_FILTER_PREFIX_* without tracking the false positive rate. Perhaps confusingly, when prefix filters are used for point queries, the stats are currently going into the non-PREFIX tickers. This change does several things: * Introduce new stat tickers for seeks and related filtering, \*LEVEL_SEEK\* * Most importantly, allows us to see opportunities for range filtering. Specifically, we can count how many times a seek in an SST file accesses at least one data block, and how many times at least one value() is then accessed. If a data block was accessed but no value(), we can generally assume that the key(s) seen was(were) not of interest so could have been filtered with the right kind of filter, avoiding the data block access. * We can get the same level of detail when a filter (for now, prefix Bloom/ribbon) is used, or not. Specifically, we can infer a false positive rate for prefix filters (not available before) from the seek "false positive" rate: when a data block is accessed but no value() is called. (There can be other explanations for a seek false positive, but in typical iterator usage it would indicate a filter false positive.) * For efficiency, I wanted to avoid making additional calls to the prefix extractor (or key comparisons, etc.), which would be required if we wanted to more precisely detect filter false positives. I believe that instrumenting value() is the best balance of efficiency vs. accurately measuring what we are often interested in. * The stats are divided between last level and non-last levels, to help understand potential tiered storage use cases. * The old BLOOM_FILTER_PREFIX_* stats have a different meaning: no longer referring to iterators but to point queries using prefix filters. BLOOM_FILTER_PREFIX_TRUE_POSITIVE is added for computing the prefix false positive rate on point queries, which can be due to filter false positives as well as different keys with the same prefix. * Similarly, the non-PREFIX BLOOM_FILTER stats are now for whole key filtering only. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11460 Test Plan: unit tests updated, including updating many to pop the stat value since last read to improve test readability and maintainability. Performance test shows a consistent small improvement with these changes, both with clang and with gcc. CPU profile indicates that RecordTick is using less CPU, and this makes sense at least for a high filter miss rate. Before, we were recording two ticks per filter miss in iterators (CHECKED & USEFUL) and now recording just one (FILTERED). Create DB with ``` TEST_TMPDIR=/dev/shm ./db_bench -benchmarks=fillrandom -num=10000000 -disable_wal=1 -write_buffer_size=30000000 -bloom_bits=8 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=0 -prefix_size=8 ``` And run simultaneous before&after with ``` TEST_TMPDIR=/dev/shm ./db_bench -readonly -benchmarks=seekrandom[-X1000] -num=10000000 -bloom_bits=8 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=0 -prefix_size=8 -seek_nexts=1 -duration=20 -seed=43 -threads=8 -cache_size=1000000000 -statistics ``` Before: seekrandom [AVG 275 runs] : 189680 (± 222) ops/sec; 18.4 (± 0.0) MB/sec After: seekrandom [AVG 275 runs] : 197110 (± 208) ops/sec; 19.1 (± 0.0) MB/sec Reviewed By: ajkr Differential Revision: D46029177 Pulled By: pdillinger fbshipit-source-id: cdace79a2ea548d46c5900b068c5b7c3a02e5822		2 years ago
..
blob	Change internal headers with duplicate names (#11408)	2 years ago
compaction	Remove wait_unscheduled from waitForCompact internal API (#11443)	2 years ago
db_impl	Support Clip DB to KeyRange (#11379)	2 years ago
wide	Add a new MultiGetEntity API (#11222)	2 years ago
arena_wrapped_db_iter.cc	Ignore async_io ReadOption if FileSystem doesn't support it (#11296)	2 years ago
arena_wrapped_db_iter.h	Ran clang-format on db/ directory (#10910)	2 years ago
builder.cc	Record and use the tail size to prefetch table tail (#11406)	2 years ago
builder.h	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288)	2 years ago
c.cc	Block fetch CPU time counters in perf context (#11342)	2 years ago
c_test.c	Add in-transaction multi-get-for-update to the C interface (#11107)	2 years ago
column_family.cc	Support compacting files to different temperatures in FIFO compaction (#11428)	2 years ago
column_family.h	New stat rocksdb.{cf|db}-write-stall-stats exposed in a structural way (#11300)	2 years ago
column_family_test.cc	Remove RocksDB LITE (#11147)	2 years ago
compact_files_test.cc	Drain unnecessary levels when `level_compaction_dynamic_level_bytes=true` (#11340)	2 years ago
comparator_db_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
convenience.cc	Block per key-value checksum (#11287)	2 years ago
corruption_test.cc	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288)	2 years ago
cuckoo_table_db_test.cc	Remove RocksDB LITE (#11147)	2 years ago
db_basic_test.cc	Ensure VerifyFileChecksums reads don't exceed readahead_size (#11328)	2 years ago
db_block_cache_test.cc	Add hash_seed to Caches (#11391)	2 years ago
db_bloom_filter_test.cc	Much better stats for seeks and prefix filtering (#11460)	2 years ago
db_clip_test.cc	Support Clip DB to KeyRange (#11379)	2 years ago
db_compaction_filter_test.cc	Always allow L0->L1 trivial move during manual compaction (#11375)	2 years ago
db_compaction_test.cc	Remove wait_unscheduled from waitForCompact internal API (#11443)	2 years ago
db_dynamic_level_test.cc	Remove RocksDB LITE (#11147)	2 years ago
db_encryption_test.cc	Remove RocksDB LITE (#11147)	2 years ago
db_filesnapshot.cc	Fix bug of prematurely excluded CF in atomic flush contains unflushed data that should've been included in the atomic flush (#11148)	2 years ago
db_flush_test.cc	Fix bug of prematurely excluded CF in atomic flush contains unflushed data that should've been included in the atomic flush (#11148)	2 years ago
db_info_dumper.cc	Ran clang-format on db/ directory (#10910)	2 years ago
db_info_dumper.h	Add a DB Session ID (#6959)	4 years ago
db_inplace_update_test.cc	Fix in-place updates for value types other than kTypeValue (#10254)	2 years ago
db_io_failure_test.cc	Remove RocksDB LITE (#11147)	2 years ago
db_iter.cc	DBIter::FindValueForCurrentKey: remove unused Status s (#11394)	2 years ago
db_iter.h	Fix backward iteration issue when user defined timestamp is enabled in BlobDB (#11258)	2 years ago
db_iter_stress_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
db_iter_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
db_iterator_test.cc	Change internal headers with duplicate names (#11408)	2 years ago
db_kv_checksum_test.cc	Print stack traces on frozen tests in CI (#10828)	2 years ago
db_log_iter_test.cc	Remove RocksDB LITE (#11147)	2 years ago
db_logical_block_size_cache_test.cc	Remove RocksDB LITE (#11147)	2 years ago
db_memtable_test.cc	Add support for wide-column point lookups (#10540)	2 years ago
db_merge_operand_test.cc	Remove RocksDB LITE (#11147)	2 years ago
db_merge_operator_test.cc	Better support for merge operation with data block hash index (#11356)	2 years ago
db_options_test.cc	Delete temp OPTIONS file on failure to write it (#11423)	2 years ago
db_properties_test.cc	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288)	2 years ago
db_range_del_test.cc	Refactor AddRangeDels() + consider range tombstone during compaction file cutting (#11113)	2 years ago
db_rate_limiter_test.cc	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288)	2 years ago
db_readonly_with_timestamp_test.cc	Remove RocksDB LITE (#11147)	2 years ago
db_secondary_test.cc	Remove RocksDB LITE (#11147)	2 years ago
db_sst_test.cc	Remove wait_unscheduled from waitForCompact internal API (#11443)	2 years ago
db_statistics_test.cc	Add block checksum mismatch ticker stat (#11438)	2 years ago
db_table_properties_test.cc	Remove RocksDB LITE (#11147)	2 years ago
db_tailing_iter_test.cc	Remove RocksDB LITE (#11147)	2 years ago
db_test.cc	Compatibility step for separating BlockCache and GeneralCache APIs (#11450)	2 years ago
db_test2.cc	Much better stats for seeks and prefix filtering (#11460)	2 years ago
db_test_util.cc	Fix flaky test `DBTestUniversalManualCompactionOutputPathId.ManualCompactionOutputPathId` (#11412)	2 years ago
db_test_util.h	Much better stats for seeks and prefix filtering (#11460)	2 years ago
db_universal_compaction_test.cc	Fix flaky test `DBTestUniversalManualCompactionOutputPathId.ManualCompactionOutputPathId` (#11412)	2 years ago
db_wal_test.cc	Delete empty WAL files on reopen (#11409)	2 years ago
db_with_timestamp_basic_test.cc	Filter table files by timestamp: Get operator (#11332)	2 years ago
db_with_timestamp_compaction_test.cc	Remove RocksDB LITE (#11147)	2 years ago
db_with_timestamp_test_util.cc	Add timestamp support to DBImplReadOnly (#10004)	3 years ago
db_with_timestamp_test_util.h	Add timestamp support to DBImplReadOnly (#10004)	3 years ago
db_write_buffer_manager_test.cc	Add `SetAllowStall()` (#11335)	2 years ago
db_write_test.cc	Deflake DBWriteTest.LockWALInEffect (#11382)	2 years ago
dbformat.cc	Remove copying of range tombstones keys in iterator (#10878)	2 years ago
dbformat.h	Refactor AddRangeDels() + consider range tombstone during compaction file cutting (#11113)	2 years ago
dbformat_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
deletefile_test.cc	Remove RocksDB LITE (#11147)	2 years ago
error_handler.cc	Remove RocksDB LITE (#11147)	2 years ago
error_handler.h	Ran clang-format on db/ directory (#10910)	2 years ago
error_handler_fs_test.cc	Remove wait_unscheduled from waitForCompact internal API (#11443)	2 years ago
event_helpers.cc	Remove FactoryFunc from LoadXXXObject (#11203)	2 years ago
event_helpers.h	Remove RocksDB LITE (#11147)	2 years ago
experimental.cc	Record and use the tail size to prefetch table tail (#11406)	2 years ago
external_sst_file_basic_test.cc	Clarify `SstFileWriter::DeleteRange()` ordering requirements (#11390)	2 years ago
external_sst_file_ingestion_job.cc	Record and use the tail size to prefetch table tail (#11406)	2 years ago
external_sst_file_ingestion_job.h	Add missing range conflict check between file ingestion and RefitLevel() (#10988)	2 years ago
external_sst_file_test.cc	Remove wait_unscheduled from waitForCompact internal API (#11443)	2 years ago
fault_injection_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
file_indexer.cc	Ran clang-format on db/ directory (#10910)	2 years ago
file_indexer.h	Ran clang-format on db/ directory (#10910)	2 years ago
file_indexer_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
filename_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
flush_job.cc	Record and use the tail size to prefetch table tail (#11406)	2 years ago
flush_job.h	Remove RocksDB LITE (#11147)	2 years ago
flush_job_test.cc	Remove RocksDB LITE (#11147)	2 years ago
flush_scheduler.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
flush_scheduler.h	Include C++ standard library headers instead of C compatibility headers (#8068)	4 years ago
forward_iterator.cc	Block per key-value checksum (#11287)	2 years ago
forward_iterator.h	Ignore async_io ReadOption if FileSystem doesn't support it (#11296)	2 years ago
forward_iterator_bench.cc	Remove RocksDB LITE (#11147)	2 years ago
history_trimming_iterator.h	Refactor AddRangeDels() + consider range tombstone during compaction file cutting (#11113)	2 years ago
import_column_family_job.cc	Record and use the tail size to prefetch table tail (#11406)	2 years ago
import_column_family_job.h	Support range deletion tombstones in `CreateColumnFamilyWithImport` (#11252)	2 years ago
import_column_family_test.cc	Fix the wrong calculation of largest_key in import_column_family_job (#11381)	2 years ago
internal_stats.cc	Fix write stall stats dump format (#11445)	2 years ago
internal_stats.h	Add hash_seed to Caches (#11391)	2 years ago
job_context.h	Remove RocksDB LITE (#11147)	2 years ago
kv_checksum.h	Block per key-value checksum (#11287)	2 years ago
listener_test.cc	Change internal headers with duplicate names (#11408)	2 years ago
log_format.h	Add support in log writer and reader for a user-defined timestamp size record (#11433)	2 years ago
log_reader.cc	Add support in log writer and reader for a user-defined timestamp size record (#11433)	2 years ago
log_reader.h	Add support in log writer and reader for a user-defined timestamp size record (#11433)	2 years ago
log_test.cc	Add support in log writer and reader for a user-defined timestamp size record (#11433)	2 years ago
log_writer.cc	Add support in log writer and reader for a user-defined timestamp size record (#11433)	2 years ago
log_writer.h	Add support in log writer and reader for a user-defined timestamp size record (#11433)	2 years ago
logs_with_prep_tracker.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
logs_with_prep_tracker.h	Ran clang-format on db/ directory (#10910)	2 years ago
lookup_key.h	Ran clang-format on db/ directory (#10910)	2 years ago
malloc_stats.cc	Remove RocksDB LITE (#11147)	2 years ago
malloc_stats.h	Remove RocksDB LITE (#11147)	2 years ago
manual_compaction_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
memtable.cc	Change internal headers with duplicate names (#11408)	2 years ago
memtable.h	Block per key-value checksum (#11287)	2 years ago
memtable_list.cc	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288)	2 years ago
memtable_list.h	Fix memtable-only iterator regression (#10705)	2 years ago
memtable_list_test.cc	Remove RocksDB LITE (#11147)	2 years ago
merge_context.h	Ran clang-format on db/ directory (#10910)	2 years ago
merge_helper.cc	Change internal headers with duplicate names (#11408)	2 years ago
merge_helper.h	Add API to limit blast radius of merge operator failure (#11092)	2 years ago
merge_helper_test.cc	Basic Support for Merge with user-defined timestamp (#10819)	2 years ago
merge_operator.cc	Ran clang-format on db/ directory (#10910)	2 years ago
merge_test.cc	Remove RocksDB LITE (#11147)	2 years ago
obsolete_files_test.cc	Remove RocksDB LITE (#11147)	2 years ago
options_file_test.cc	Remove RocksDB LITE (#11147)	2 years ago
output_validator.cc	Cleanup includes in dbformat.h (#8930)	3 years ago
output_validator.h	Cleanup includes in dbformat.h (#8930)	3 years ago
perf_context_test.cc	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288)	2 years ago
periodic_task_scheduler.cc	Remove RocksDB LITE (#11147)	2 years ago
periodic_task_scheduler.h	Remove RocksDB LITE (#11147)	2 years ago
periodic_task_scheduler_test.cc	Remove RocksDB LITE (#11147)	2 years ago
pinned_iterators_manager.h	Avoid allocations/copies for large `GetMergeOperands()` results (#10458)	2 years ago
plain_table_db_test.cc	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288)	2 years ago
post_memtable_callback.h	Snapshots with user-specified timestamps (#9879)	2 years ago
pre_release_callback.h	Fix and detect headers with missing dependencies (#8893)	3 years ago
prefix_test.cc	Remove RocksDB LITE (#11147)	2 years ago
range_del_aggregator.cc	Improve documentation for MergingIterator (#11161)	2 years ago
range_del_aggregator.h	Improve documentation for MergingIterator (#11161)	2 years ago
range_del_aggregator_bench.cc	Improve FragmentTombstones() speed by lazily initializing `seq_set_` (#10848)	2 years ago
range_del_aggregator_test.cc	Refactor AddRangeDels() + consider range tombstone during compaction file cutting (#11113)	2 years ago
range_tombstone_fragmenter.cc	Improve FragmentTombstones() speed by lazily initializing `seq_set_` (#10848)	2 years ago
range_tombstone_fragmenter.h	Refactor AddRangeDels() + consider range tombstone during compaction file cutting (#11113)	2 years ago
range_tombstone_fragmenter_test.cc	snapshots of FragmentedRangeTombstoneList must in ascending order (#11046)	2 years ago
read_callback.h	Fix and detect headers with missing dependencies (#8893)	3 years ago
repair.cc	Record and use the tail size to prefetch table tail (#11406)	2 years ago
repair_test.cc	Remove RocksDB LITE (#11147)	2 years ago
seqno_time_test.cc	Remove wait_unscheduled from waitForCompact internal API (#11443)	2 years ago
seqno_to_time_mapping.cc	Add option `preserve_internal_time_seconds` to preserve the time info (#10747)	2 years ago
seqno_to_time_mapping.h	Add option `preserve_internal_time_seconds` to preserve the time info (#10747)	2 years ago
snapshot_checker.h	Remove RocksDB LITE (#11147)	2 years ago
snapshot_impl.cc	Ran clang-format on db/ directory (#10910)	2 years ago
snapshot_impl.h	Ran clang-format on db/ directory (#10910)	2 years ago
table_cache.cc	Record and use the tail size to prefetch table tail (#11406)	2 years ago
table_cache.h	Block per key-value checksum (#11287)	2 years ago
table_cache_sync_and_async.h	Block per key-value checksum (#11287)	2 years ago
table_properties_collector.cc	Ran clang-format on db/ directory (#10910)	2 years ago
table_properties_collector.h	Fix an assertion failure in `TimestampTablePropertiesCollector` for empty output (#11015)	2 years ago
table_properties_collector_test.cc	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288)	2 years ago
transaction_log_impl.cc	Remove RocksDB LITE (#11147)	2 years ago
transaction_log_impl.h	Remove RocksDB LITE (#11147)	2 years ago
trim_history_scheduler.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
trim_history_scheduler.h	Ran clang-format on db/ directory (#10910)	2 years ago
version_builder.cc	Block per key-value checksum (#11287)	2 years ago
version_builder.h	Block per key-value checksum (#11287)	2 years ago
version_builder_test.cc	Record and use the tail size to prefetch table tail (#11406)	2 years ago
version_edit.cc	Record and use the tail size to prefetch table tail (#11406)	2 years ago
version_edit.h	Record and use the tail size to prefetch table tail (#11406)	2 years ago
version_edit_handler.cc	Block per key-value checksum (#11287)	2 years ago
version_edit_handler.h	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288)	2 years ago
version_edit_test.cc	Record and use the tail size to prefetch table tail (#11406)	2 years ago
version_set.cc	Support Clip DB to KeyRange (#11379)	2 years ago
version_set.h	Support Clip DB to KeyRange (#11379)	2 years ago
version_set_sync_and_async.h	Block per key-value checksum (#11287)	2 years ago
version_set_test.cc	Record and use the tail size to prefetch table tail (#11406)	2 years ago
version_util.h	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288)	2 years ago
wal_edit.cc	Do not hold mutex when write keys if not necessary (#7516)	2 years ago
wal_edit.h	Do not hold mutex when write keys if not necessary (#7516)	2 years ago
wal_edit_test.cc	Do not hold mutex when write keys if not necessary (#7516)	2 years ago
wal_manager.cc	Remove RocksDB LITE (#11147)	2 years ago
wal_manager.h	Remove RocksDB LITE (#11147)	2 years ago
wal_manager_test.cc	Remove RocksDB LITE (#11147)	2 years ago
write_batch.cc	Change internal headers with duplicate names (#11408)	2 years ago
write_batch_base.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
write_batch_internal.h	Ran clang-format on db/ directory (#10910)	2 years ago
write_batch_test.cc	Remove RocksDB LITE (#11147)	2 years ago
write_callback.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
write_callback_test.cc	Remove RocksDB LITE (#11147)	2 years ago
write_controller.cc	Revamp WriteController (#8064)	4 years ago
write_controller.h	Ran clang-format on db/ directory (#10910)	2 years ago
write_controller_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
write_stall_stats.cc	Fix initialization-order-fiasco in write_stall_stats.cc (#11355)	2 years ago
write_stall_stats.h	Fix initialization-order-fiasco in write_stall_stats.cc (#11355)	2 years ago
write_thread.cc	Ensure LockWAL() stall cleared for UnlockWAL() return (#11172)	2 years ago
write_thread.h	Ensure LockWAL() stall cleared for UnlockWAL() return (#11172)	2 years ago