rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

Peter Dillinger f9db0c6e9c Refactor block cache tracing w/improved MultiGet (#11339 ) Summary: After https://github.com/facebook/rocksdb/issues/11301, I wasn't sure whether I had regressed block cache tracing with MultiGet. Demo PR https://github.com/facebook/rocksdb/issues/11330 shows the flawed state of tracing MultiGet before my change, and based on the unit test, there was essentially no change in tracing behavior with https://github.com/facebook/rocksdb/issues/11301. This change is to leave that code and behavior better than I found it. This change is not intended to change any production behaviors except when block cache tracing is active, though might improve general read path efficiency by disabling some related tracking when such tracing is disabled. More detail on production code: * Refactoring to consolidate the construction of BlockCacheTraceRecord, and other related functionality, in block-based table reader, though it's somewhat awkward to preserve an optimization to avoid copying Slices into temporary strings in BlockCacheLookupContext. * Accurately track cache hits and misses (etc.) for each data block accessed by a MultiGet(). (Previously reported hits as misses.) * Reduced repeated checking of `block_cache_tracer_` state (by creating lookup_context only when active) for efficiency and to reduce the risk of corner case bugs where tracing is enabled or disabled for different parts of a read op. (See a TODO below) * Improved estimate calculation for num_keys_in_block (see code comment) Possible follow-up: * `XXX:` use_cache=true means double cache query? (possible double-query of block cache when allow_mmap_reads=true) * `TODO:` need more than one lookup_context here to track individual filter and index partition hits and misses * `TODO:` optimize more state checks of `block_cache_tracer_` down to `lookup_context != nullptr` * Pre-existing `XXX:` There appear to be 'break' statements above that bypass this writing of the block cache trace record * Expand test coverage (see below) Pull Request resolved: https://github.com/facebook/rocksdb/pull/11339 Test Plan: * Added a basic unit test for block cache tracing MultiGet, for now just covering one data block with two keys. * Added HitMissCountingCache to independently verify that the actual block cache trace and expected block cache trace also agree with the actual number of cache hits / misses (nothing missing or mislabeled). For now only used with MultiGet test. * Better testing of num_keys_in_block, for now just with MultiGet * Misc improvements to table_test to improve clarity, such as making it clear that certain keys are auto-inserted at the start of every test. Performance test: Testing multireadrandom as in https://github.com/facebook/rocksdb/issues/11301, except averaging over distinct runs rather than [-X30] which doesn't seem to sufficiently reset after each run to work as an independent test run. Base with revert of 11301: 3148926 ops/sec Base: 3019146 ops/sec New: 2999529 ops/sec Possibly a tiny MultiGet CPU regression with this change. We are now always allocating an additional vector for the LookupContexts. I'm still contemplating options to try to correct the regression in https://github.com/facebook/rocksdb/issues/11301. Testing readrandom: Base with revert of 11301: 2311988 Base: 2281726 New: 2299722 Possibly a tiny Get CPU improvement with this change. We are now avoiding some unnecessary LookupContext population. Reviewed By: akankshamahajan15 Differential Revision: D44557845 Pulled By: pdillinger fbshipit-source-id: b841691799d2a48fb59cc8880dc7cbb1e107ae3d		2 years ago
.circleci	Revert enabling IO uring in db_stress (#11242 )	2 years ago
.github/workflows	ci: add GitHub token permissions for workflow (#10549 )	2 years ago
buckifier	Cleanup, improve, stress test LockWAL() (#11143 )	2 years ago
build_tools	Remove platform009 and default to platform010 (#11333 )	2 years ago
cache	Re-clarify SecondaryCache API (#11316 )	2 years ago
cmake	gcc-11 and cmake related cleanup (#9286 )	3 years ago
coverage	Remove platform009 and default to platform010 (#11333 )	2 years ago
db	Better support for merge operation with data block hash index (#11356 )	2 years ago
db_stress_tool	Extend the stress test coverage of MultiGetEntity (#11336 )	2 years ago
docs	Remove docs/Gemfile.lock and update github-pages version (#11173 )	2 years ago
env	Ignore async_io ReadOption if FileSystem doesn't support it (#11296 )	2 years ago
examples	Remove RocksDB LITE (#11147 )	2 years ago
file	Ensure VerifyFileChecksums reads don't exceed readahead_size (#11328 )	2 years ago
fuzz	Add some missing headers (#10519 )	2 years ago
include/rocksdb	Filter table files by timestamp: Get operator (#11332 )	2 years ago
java	New stat rocksdb.{cf\|db}-write-stall-stats exposed in a structural way (#11300 )	2 years ago
logging	Disabling some IO error assertion in EnvLogger (#11314 )	2 years ago
memory	Remove RocksDB LITE (#11147 )	2 years ago
memtable	Add `SetAllowStall()` (#11335 )	2 years ago
microbench	Avoid allocations/copies for large `GetMergeOperands()` results (#10458 )	2 years ago
monitoring	Filter table files by timestamp: Get operator (#11332 )	2 years ago
options	Change default block cache from 8MB to 32MB (#11350 )	2 years ago
plugin	Add initial CMake support to plugin (#9214 )	3 years ago
port	Ignore async_io ReadOption if FileSystem doesn't support it (#11296 )	2 years ago
table	Refactor block cache tracing w/improved MultiGet (#11339 )	2 years ago
test_util	HyperClockCache support for SecondaryCache, with refactoring (#11301 )	2 years ago
third-party	Meta-internal folly integration with F14FastMap (#9546 )	3 years ago
tools	Change default block cache from 8MB to 32MB (#11350 )	2 years ago
trace_replay	Fix error maybe-uninitialized #11100 (#11101 )	2 years ago
util	Use user-provided ReadOptions for metadata block reads more often (#11208 )	2 years ago
utilities	HyperClockCache support for SecondaryCache, with refactoring (#11301 )	2 years ago
.clang-format	A script that automatically reformat affected lines	11 years ago
.gitignore	Git ignore .clangd/ (#10817 )	2 years ago
.lgtm.yml	Create lgtm.yml for LGTM.com C/C++ analysis (#4058 )	6 years ago
.watchmanconfig	Added .watchmanconfig file to rocksdb repo (#5593 )	5 years ago
AUTHORS	Update RocksDB Authors File	7 years ago
CMakeLists.txt	New stat rocksdb.{cf\|db}-write-stall-stats exposed in a structural way (#11300 )	2 years ago
CODE_OF_CONDUCT.md	Adopt Contributor Covenant	5 years ago
CONTRIBUTING.md	Add Code of Conduct	7 years ago
COPYING	Add GPLv2 as an alternative license.	8 years ago
DEFAULT_OPTIONS_HISTORY.md	Add Options::DisableExtraChecks, clarify force_consistency_checks (#9363 )	3 years ago
DUMP_FORMAT.md	First version of rocksdb_dump and rocksdb_undump.	10 years ago
HISTORY.md	Refactor block cache tracing w/improved MultiGet (#11339 )	2 years ago
INSTALL.md	Add instructions for installing googlebenchmark (#11282 )	2 years ago
LANGUAGE-BINDINGS.md	Add grocksdb in Go language bindings (#10498 )	2 years ago
LICENSE.Apache	Change RocksDB License	7 years ago
LICENSE.leveldb	Add back the LevelDB license file	7 years ago
Makefile	Revert to LIB_MODE=static for optimized builds (#11195 )	2 years ago
PLUGINS.md	Add pmem-rocksdb-plugin link in PLUGINs.md (#9934 )	3 years ago
README.md	Remove Travis CI (#10407 )	2 years ago
TARGETS	New stat rocksdb.{cf\|db}-write-stall-stats exposed in a structural way (#11300 )	2 years ago
USERS.md	Add Microsoft Bing as a user (#11270 )	2 years ago
Vagrantfile	Adding CentOS 7 Vagrantfile & build script	7 years ago
WINDOWS_PORT.md	Update branch name in WINDOWS_PORT.md (#8745 )	3 years ago
common.mk	Clean up variables for temporary directory (#9961 )	3 years ago
crash_test.mk	Allow a custom DB cleanup command to be passed to db_crashtest.py (#10883 )	2 years ago
issue_template.md	Add Google Group to Issue Template	5 years ago
rocksdb.pc.in	build: fix pkg-config file generation (#9953 )	2 years ago
src.mk	New stat rocksdb.{cf\|db}-write-stall-stats exposed in a structural way (#11300 )	2 years ago
thirdparty.inc	Fix build jemalloc api (#5470 )	5 years ago

README.md

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/main/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.

Questions and discussions are welcome on the RocksDB Developers Public Facebook group and email list on Google Groups.

License

RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.