rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

Andrew Kryczka f24c39ab3d Prevent corruption with parallel manual compactions and `change_level == true` (#9077 ) Summary: The bug can impact the following scenario. There must be two `CompactRange()`s, call them A and B. Compaction A must have `change_level=true`. Compactions A and B must run in parallel, and new data must be added while they run as well. Now, on to the details of the race condition. Compaction A must reach the refitting phase while B's next step is to trivial move new data (i.e., data that has been inserted behind A) down to the same level that A's refit targets (`CompactRangeOptions::target_level`). B must be unregistered (i.e., has not yet called `AddManualCompaction()` for the current `RunManualCompaction()`) while A invokes `DisableManualCompaction()`s to prepare for refitting. In the old code, B could still proceed to register a manual compaction, while A had disabled manual compaction. The next part of the race condition is B picks and schedules a trivial move while A has released the lock in refitting phase in order to persist the LSM state change (i.e., the log phase of `LogAndApply()`). That way, B does not see the refitted data when picking a trivial-move compaction. So it is susceptible to picking one that overlaps. Finally, B executes the picked trivial-move compaction. Trivial-move compactions are special in that they never check whether manual compaction is disabled. So the picked compaction causing overlap ends up being applied, leading to LSM corruption if `force_consistency_checks=false`, or entering read-only mode with `Status::Corruption` if `force_consistency_checks=true` (the default). The fix is just to prevent B from registering itself in `RunManualCompaction()` while manual compactions are disabled, consequently preventing any trivial move or other compaction from being picked/scheduled. Thanks to siying for finding the bug. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9077 Test Plan: The test does not go all the way in exposing the bug because it requires a compaction to be picked/scheduled while logging LSM state change for RefitLevel(). But the fix is to make such a compaction not picked/scheduled in the first place, so any repro of that scenario would end up hanging RefitLevel() logging. So instead I just verified no such compaction is registered in the scenario where `RefitLevel()` disables manual compactions. Reviewed By: siying Differential Revision: D31921908 Pulled By: ajkr fbshipit-source-id: 9bb5d0e847ad428211227f40830c685c209fbecb		3 years ago
.circleci	Fix gcc-11 compile error (#9043 )	3 years ago
.github/workflows	Add (& fix) some simple source code checks (#8821 )	3 years ago
buckifier	Modify script which generates TARGETS (#8366 )	3 years ago
build_tools	Make format-diff.sh locale-independent (#9079 )	3 years ago
cache	Add lowest_used_cache_tier to ImmutableDBOptions to enable or disable Secondary Cache (#9050 )	3 years ago
cmake	Add `find_dependency()` in cmake config file. (#6791 )	5 years ago
coverage	Remove asan_symbolize.py for internal asan build (#8737 )	3 years ago
db	Prevent corruption with parallel manual compactions and `change_level == true` (#9077 )	3 years ago
db_stress_tool	Fix stress/crash test handling of SST unique IDs (#9054 )	3 years ago
docs	Misc doc fixes (#8983 )	3 years ago
env	Experimental support for SST unique IDs (#8990 )	3 years ago
examples	Add (& fix) some simple source code checks (#8821 )	3 years ago
file	Add (Live)FileStorageInfo API (#8968 )	3 years ago
fuzz	Make EventListener into a Customizable Class (#8473 )	3 years ago
hdfs	fix build with 'USE_HDFS' on windows (#6950 )	4 years ago
include/rocksdb	Clarify caching behavior for index and filter partitions (#9068 )	3 years ago
java	Introduce allowStall option for write buffer manager constructor (#9076 )	3 years ago
logging	Make SystemClock into a Customizable Class (#8636 )	3 years ago
memory	Fix and detect headers with missing dependencies (#8893 )	3 years ago
memtable	Fix race in WriteBufferManager (#9009 )	3 years ago
microbench	Add micro-benchmark support (#8493 )	3 years ago
monitoring	Add support for building on s390x platform (#8962 )	3 years ago
options	Incremental Space Amp Compactions in Universal Style (#8655 )	3 years ago
plugin	Makefile support to statically link external plugin code (#7918 )	4 years ago
port	Fix incorrect order of comments in win_thread.cc (#9033 )	3 years ago
table	Clarify caching behavior for index and filter partitions (#9068 )	3 years ago
test_util	SyncPoint::Process thrashes heap ... fix it (#9023 )	3 years ago
third-party	Add support for building on s390x platform (#8962 )	3 years ago
tools	Incremental Space Amp Compactions in Universal Style (#8655 )	3 years ago
trace_replay	Cleanup includes in dbformat.h (#8930 )	3 years ago
util	Add support for building on s390x platform (#8962 )	3 years ago
utilities	Add support for building on s390x platform (#8962 )	3 years ago
.clang-format	A script that automatically reformat affected lines	11 years ago
.gitignore	gitignore cmake-build-* for CLion integration (#7933 )	4 years ago
.lgtm.yml	Create lgtm.yml for LGTM.com C/C++ analysis (#4058 )	6 years ago
.travis.yml	Add support for building on s390x platform (#8962 )	3 years ago
.watchmanconfig	Added .watchmanconfig file to rocksdb repo (#5593 )	5 years ago
AUTHORS	Update RocksDB Authors File	7 years ago
CMakeLists.txt	Add support for building on s390x platform (#8962 )	3 years ago
CODE_OF_CONDUCT.md	Adopt Contributor Covenant	5 years ago
CONTRIBUTING.md	Add Code of Conduct	7 years ago
COPYING	Add GPLv2 as an alternative license.	8 years ago
DEFAULT_OPTIONS_HISTORY.md	options.delayed_write_rate use the rate of rate_limiter by default.	8 years ago
DUMP_FORMAT.md	First version of rocksdb_dump and rocksdb_undump.	10 years ago
HISTORY.md	Prevent corruption with parallel manual compactions and `change_level == true` (#9077 )	3 years ago
INSTALL.md	Update installation instructions (#8158 )	4 years ago
LANGUAGE-BINDINGS.md	Update branch name to "main" in README/LANGUAGE_BINDINGS (#8727 )	3 years ago
LICENSE.Apache	Change RocksDB License	7 years ago
LICENSE.leveldb	Add back the LevelDB license file	7 years ago
Makefile	Add support for building on s390x platform (#8962 )	3 years ago
PLUGINS.md	Add ZenFS to plugin list (#8218 )	4 years ago
README.md	Update branch name to "main" in README/LANGUAGE_BINDINGS (#8727 )	3 years ago
ROCKSDB_LITE.md	Fix some typos in comments and docs.	7 years ago
TARGETS	Experimental support for SST unique IDs (#8990 )	3 years ago
USERS.md	Update USERS.md (#8923 )	3 years ago
Vagrantfile	Adding CentOS 7 Vagrantfile & build script	7 years ago
WINDOWS_PORT.md	Update branch name in WINDOWS_PORT.md (#8745 )	3 years ago
appveyor.yml	Remove 2019 from appveyor (#7038 )	4 years ago
defs.bzl	Make testpilot recognize that these tests have coverage instrumentation	5 years ago
issue_template.md	Add Google Group to Issue Template	5 years ago
src.mk	Experimental support for SST unique IDs (#8990 )	3 years ago
thirdparty.inc	Fix build jemalloc api (#5470 )	5 years ago

README.md

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/main/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.

Design discussions are conducted in https://www.facebook.com/groups/rocksdb.dev/ and https://rocksdb.slack.com/

License

RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.