rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

Cheng Chang efe827baf0 Always track WAL obsoletion (#7759 ) Summary: Currently, when a WAL becomes obsolete after flushing, if VersionSet::WalSet does not contain the WAL, we do not track the WAL obsoletion event in MANIFEST. But consider this case: * WAL 10 is synced, a VersionEdit is LogAndApplied to MANIFEST to log this WAL addition event, but the VersionEdit is not applied to WalSet yet since its corresponding ManifestWriter is still pending in the write queue; * Since the above ManifestWriter is blocking, the LogAndApply will block on a conditional variable and release the db mutex, so another LogAndApply can proceed to enqueue other VersionEdits concurrently; * Now flush happens, and WAL 10 becomes obsolete, although WalSet does not contain WAL 10 yet, we should call LogAndApply to enqueue a VersionEdit to indicate the obsoletion of WAL 10; * otherwise, when the queued edit indicating WAL 10 addition is logged to MANIFEST, and DB crashes and reopens, the WAL 10 might have been removed from disk, but it still exists in MANIFEST. This PR changes the behavior to: always `LogAndApply` any WAL addition or obsoletion event, without considering the order issues caused by concurrency, but when applying the edits to `WalSet`, do not add the WALs if they are already obsolete. In this approach, the logical events of WAL addition and obsoletion are always tracked in MANIFEST, so we can inspect the MANIFEST and know all the previous WAL events, but we choose to ignore certain events due to the concurrency issues such as the case above, or the case in https://github.com/facebook/rocksdb/pull/7725. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7759 Test Plan: make check Reviewed By: pdillinger Differential Revision: D25423089 Pulled By: cheng-chang fbshipit-source-id: 9cb9a7fbc1875bf954f2a42f9b6cfd6d49a7b21c		4 years ago
.circleci	Migrate away from broken macos on Travis (#7745 )	4 years ago
.github/workflows	Update clang-format-diff.py (#7609 )	4 years ago
buckifier	Make the TARGETS file Starlark compliant (#7743 )	4 years ago
build_tools	RocksJava static lib dependencies should support MacOS 10.12+ (#7683 )	4 years ago
cache	Fix typos in comments (#7687 )	4 years ago
cmake	Add `find_dependency()` in cmake config file. (#6791 )	5 years ago
coverage	Find the correct gcov (#6904 )	5 years ago
db	Always track WAL obsoletion (#7759 )	4 years ago
db_stress_tool	Experimental (production candidate) SST schema for Ribbon filter (#7658 )	4 years ago
docs	Update github-pages to v207 (#7235 )	4 years ago
env	Use default FileSystem in GenerateUniqueId (#7672 )	4 years ago
examples	Bring the Configurable options together (#5753 )	4 years ago
file	Remove unused includes (#7604 )	4 years ago
fuzz	OSS-Fuzz integration and db_fuzzer (#7674 )	4 years ago
hdfs	fix build with 'USE_HDFS' on windows (#6950 )	5 years ago
include/rocksdb	Range Locking: Allow different LockManagers, add Range Lock definitions (#7443 )	4 years ago
java	Fix a recovery corner case (#7621 )	4 years ago
logging	Remove unused includes (#7604 )	4 years ago
memory	slightly improve jemalloc allocator API header (#7592 )	4 years ago
memtable	Test for LoadLatestOptions (#7554 )	4 years ago
monitoring	Remove unused includes (#7604 )	4 years ago
options	Hack to load OPTIONS file for read_amp_bytes_per_bit (#7659 )	4 years ago
port	Warn about practically unfixable TSAN warnings in stack trace (#7723 )	4 years ago
table	Add blob support to DBIter (#7731 )	4 years ago
test_util	Port corruption test to use custom env (#7699 )	4 years ago
third-party	Fix Compilation on ppc64le using Clang 11 (#7713 )	4 years ago
tools	add 6.15.fb to check_format_compatible.sh (#7738 )	4 years ago
trace_replay	Genericize and clean up FastRange (#7436 )	4 years ago
util	Fix build on FreeBSD/powerpc64(le) (#7732 )	4 years ago
utilities	LockTree library, originally from PerconaFT (#7753 )	4 years ago
.clang-format	A script that automatically reformat affected lines	11 years ago
.gitignore	Fuzzing RocksDB (#7685 )	4 years ago
.lgtm.yml	Create lgtm.yml for LGTM.com C/C++ analysis (#4058 )	7 years ago
.travis.yml	Migrate away from broken macos on Travis (#7745 )	4 years ago
.watchmanconfig	Added .watchmanconfig file to rocksdb repo (#5593 )	6 years ago
AUTHORS	Update RocksDB Authors File	7 years ago
CMakeLists.txt	LockTree library, originally from PerconaFT (#7753 )	4 years ago
CODE_OF_CONDUCT.md	Adopt Contributor Covenant	5 years ago
CONTRIBUTING.md	Add Code of Conduct	7 years ago
COPYING	Add GPLv2 as an alternative license.	8 years ago
DEFAULT_OPTIONS_HISTORY.md	options.delayed_write_rate use the rate of rate_limiter by default.	8 years ago
DUMP_FORMAT.md	First version of rocksdb_dump and rocksdb_undump.	10 years ago
HISTORY.md	Write min_log_number_to_keep to MANIFEST during atomic flush under 2 phase commit (#7570 )	4 years ago
INSTALL.md	Update the version of the dependencies used by the RocksJava static build (#4761 )	6 years ago
LANGUAGE-BINDINGS.md	Add RestoreDBFromLatestBackup to C API, add new C# package (#7092 )	5 years ago
LICENSE.Apache	Change RocksDB License	8 years ago
LICENSE.leveldb	Add back the LevelDB license file	8 years ago
Makefile	Add further tests to ASSERT_STATUS_CHECKED (1) (#7679 )	4 years ago
README.md	Fix the CI badge for ppc64le Jenkins (#7561 )	4 years ago
ROCKSDB_LITE.md	Fix some typos in comments and docs.	7 years ago
TARGETS	LockTree library, originally from PerconaFT (#7753 )	4 years ago
USERS.md	add ArangoDB to USERS.md, and fix typos in that file (#7675 )	4 years ago
Vagrantfile	Adding CentOS 7 Vagrantfile & build script	7 years ago
WINDOWS_PORT.md	#5145 , rename port/dirent.h to port/port_dirent.h to avoid compile err when use port dir as header dir output (#5152 )	6 years ago
appveyor.yml	Remove 2019 from appveyor (#7038 )	5 years ago
defs.bzl	Make testpilot recognize that these tests have coverage instrumentation	5 years ago
issue_template.md	Add Google Group to Issue Template	5 years ago
src.mk	LockTree library, originally from PerconaFT (#7753 )	4 years ago
thirdparty.inc	Fix build jemalloc api (#5470 )	6 years ago

README.md

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/master/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.

Design discussions are conducted in https://www.facebook.com/groups/rocksdb.dev/ and https://rocksdb.slack.com/

License

RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.