rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

Peter Dillinger 39f5846ec7 Much better stats for seeks and prefix filtering (#11460 ) Summary: We want to know more about opportunities for better range filters, and the effectiveness of our own range filters. Currently the stats are very limited, essentially logging just hits and misses against prefix filters for range scans in BLOOM_FILTER_PREFIX_* without tracking the false positive rate. Perhaps confusingly, when prefix filters are used for point queries, the stats are currently going into the non-PREFIX tickers. This change does several things: * Introduce new stat tickers for seeks and related filtering, \LEVEL_SEEK\ * Most importantly, allows us to see opportunities for range filtering. Specifically, we can count how many times a seek in an SST file accesses at least one data block, and how many times at least one value() is then accessed. If a data block was accessed but no value(), we can generally assume that the key(s) seen was(were) not of interest so could have been filtered with the right kind of filter, avoiding the data block access. * We can get the same level of detail when a filter (for now, prefix Bloom/ribbon) is used, or not. Specifically, we can infer a false positive rate for prefix filters (not available before) from the seek "false positive" rate: when a data block is accessed but no value() is called. (There can be other explanations for a seek false positive, but in typical iterator usage it would indicate a filter false positive.) * For efficiency, I wanted to avoid making additional calls to the prefix extractor (or key comparisons, etc.), which would be required if we wanted to more precisely detect filter false positives. I believe that instrumenting value() is the best balance of efficiency vs. accurately measuring what we are often interested in. * The stats are divided between last level and non-last levels, to help understand potential tiered storage use cases. * The old BLOOM_FILTER_PREFIX_* stats have a different meaning: no longer referring to iterators but to point queries using prefix filters. BLOOM_FILTER_PREFIX_TRUE_POSITIVE is added for computing the prefix false positive rate on point queries, which can be due to filter false positives as well as different keys with the same prefix. * Similarly, the non-PREFIX BLOOM_FILTER stats are now for whole key filtering only. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11460 Test Plan: unit tests updated, including updating many to pop the stat value since last read to improve test readability and maintainability. Performance test shows a consistent small improvement with these changes, both with clang and with gcc. CPU profile indicates that RecordTick is using less CPU, and this makes sense at least for a high filter miss rate. Before, we were recording two ticks per filter miss in iterators (CHECKED & USEFUL) and now recording just one (FILTERED). Create DB with ``` TEST_TMPDIR=/dev/shm ./db_bench -benchmarks=fillrandom -num=10000000 -disable_wal=1 -write_buffer_size=30000000 -bloom_bits=8 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=0 -prefix_size=8 ``` And run simultaneous before&after with ``` TEST_TMPDIR=/dev/shm ./db_bench -readonly -benchmarks=seekrandom[-X1000] -num=10000000 -bloom_bits=8 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=0 -prefix_size=8 -seek_nexts=1 -duration=20 -seed=43 -threads=8 -cache_size=1000000000 -statistics ``` Before: seekrandom [AVG 275 runs] : 189680 (± 222) ops/sec; 18.4 (± 0.0) MB/sec After: seekrandom [AVG 275 runs] : 197110 (± 208) ops/sec; 19.1 (± 0.0) MB/sec Reviewed By: ajkr Differential Revision: D46029177 Pulled By: pdillinger fbshipit-source-id: cdace79a2ea548d46c5900b068c5b7c3a02e5822		2 years ago
.circleci	Revert enabling IO uring in db_stress (#11242 )	2 years ago
.github/workflows	ci: add GitHub token permissions for workflow (#10549 )	2 years ago
buckifier	Fix duplicate symbols in linking with buck2 (#11421 )	2 years ago
build_tools	Simplify detection of x86 CPU features (#11419 )	2 years ago
cache	Compatibility step for separating BlockCache and GeneralCache APIs (#11450 )	2 years ago
cmake	gcc-11 and cmake related cleanup (#9286 )	3 years ago
coverage	Remove platform009 and default to platform010 (#11333 )	2 years ago
db	Much better stats for seeks and prefix filtering (#11460 )	2 years ago
db_stress_tool	Improve comment of ExpectedValue in db stress (#11456 )	2 years ago
docs	Remove docs/Gemfile.lock and update github-pages version (#11173 )	2 years ago
env	Change internal headers with duplicate names (#11408 )	2 years ago
examples	Remove RocksDB LITE (#11147 )	2 years ago
file	Add `rocksdb.file.read.db.open.micros` (#11455 )	2 years ago
fuzz	Block per key-value checksum (#11287 )	2 years ago
include/rocksdb	Much better stats for seeks and prefix filtering (#11460 )	2 years ago
java	Add `rocksdb.file.read.db.open.micros` (#11455 )	2 years ago
logging	Disabling some IO error assertion in EnvLogger (#11314 )	2 years ago
memory	Change internal headers with duplicate names (#11408 )	2 years ago
memtable	remove redundant move (#11418 )	2 years ago
microbench	Remove wait_unscheduled from waitForCompact internal API (#11443 )	2 years ago
monitoring	Much better stats for seeks and prefix filtering (#11460 )	2 years ago
options	Change internal headers with duplicate names (#11408 )	2 years ago
plugin	Add initial CMake support to plugin (#9214 )	3 years ago
port	Simplify detection of x86 CPU features (#11419 )	2 years ago
table	Much better stats for seeks and prefix filtering (#11460 )	2 years ago
test_util	Add hash_seed to Caches (#11391 )	2 years ago
third-party	fix optimization-disabled test builds with platform010 (#11361 )	2 years ago
tools	Change internal headers with duplicate names (#11408 )	2 years ago
trace_replay	Fix error maybe-uninitialized #11100 (#11101 )	2 years ago
util	Much better stats for seeks and prefix filtering (#11460 )	2 years ago
utilities	Remove wait_unscheduled from waitForCompact internal API (#11443 )	2 years ago
.clang-format	A script that automatically reformat affected lines	11 years ago
.gitignore	remove redundant move (#11418 )	2 years ago
.lgtm.yml	Create lgtm.yml for LGTM.com C/C++ analysis (#4058 )	7 years ago
.watchmanconfig	Added .watchmanconfig file to rocksdb repo (#5593 )	6 years ago
AUTHORS	Update RocksDB Authors File	7 years ago
CMakeLists.txt	Support Clip DB to KeyRange (#11379 )	2 years ago
CODE_OF_CONDUCT.md	Adopt Contributor Covenant	5 years ago
CONTRIBUTING.md	Add Code of Conduct	7 years ago
COPYING	Add GPLv2 as an alternative license.	8 years ago
DEFAULT_OPTIONS_HISTORY.md	Add Options::DisableExtraChecks, clarify force_consistency_checks (#9363 )	3 years ago
DUMP_FORMAT.md	First version of rocksdb_dump and rocksdb_undump.	10 years ago
HISTORY.md	Much better stats for seeks and prefix filtering (#11460 )	2 years ago
INSTALL.md	Simplify detection of x86 CPU features (#11419 )	2 years ago
LANGUAGE-BINDINGS.md	Add grocksdb in Go language bindings (#10498 )	2 years ago
LICENSE.Apache	Change RocksDB License	8 years ago
LICENSE.leveldb	Add back the LevelDB license file	8 years ago
Makefile	Support Clip DB to KeyRange (#11379 )	2 years ago
PLUGINS.md	Added encryption plugin based on Intel open-source ipp-crypto library (#11429 )	2 years ago
README.md	Remove deprecated integration tests from README.md (#11354 )	2 years ago
TARGETS	Support Clip DB to KeyRange (#11379 )	2 years ago
USERS.md	Add more users (#11369 )	2 years ago
Vagrantfile	Adding CentOS 7 Vagrantfile & build script	7 years ago
WINDOWS_PORT.md	Update branch name in WINDOWS_PORT.md (#8745 )	3 years ago
common.mk	Clean up variables for temporary directory (#9961 )	3 years ago
crash_test.mk	Allow a custom DB cleanup command to be passed to db_crashtest.py (#10883 )	2 years ago
issue_template.md	Add Google Group to Issue Template	5 years ago
rocksdb.pc.in	build: fix pkg-config file generation (#9953 )	3 years ago
src.mk	Support Clip DB to KeyRange (#11379 )	2 years ago
thirdparty.inc	Fix build jemalloc api (#5470 )	6 years ago

README.md

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/main/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.

Questions and discussions are welcome on the RocksDB Developers Public Facebook group and email list on Google Groups.

License

RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.