rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

Levi Tamasi 78e291b17e Improve consistency checks in VersionBuilder (#6901 ) Summary: The patch cleans up the code and improves the consistency checks around adding/deleting table files in `VersionBuilder`. Namely, it makes the checks stricter and improves them in the following ways: 1) A table file can now only be deleted from the LSM tree using the level it resides on. Earlier, there was some unnecessary wiggle room for trivially moved files (they could be deleted using a lower level number than the actual one). 2) A table file cannot be added to the tree if it is already present in the tree on any level (not just the target level). The earlier code only had an assertion (which is a no-op in release builds) that the newly added file is not already present on the target level. 3) The above consistency checks around state transitions are now mandatory, as opposed to the earlier `CheckConsistencyForDeletes`, which was a no-op in release mode unless `force_consistency_checks` was set to `true`. The rationale here is that assuming that the initial state is consistent, a valid transition leads to a next state that is also consistent; however, an invalid transition offers no such guarantee. Hence it makes sense to validate the transitions unconditionally, and save `force_consistency_checks` for the paranoid checks that re-validate the entire state. 4) The new checks build on the mechanism introduced in https://github.com/facebook/rocksdb/pull/6862, which enables us to efficiently look up the location (level and position within level) of files in a `Version` by file number. This makes the consistency checks much more efficient than the earlier `CheckConsistencyForDeletes`, which essentially performed a linear search. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6901 Test Plan: Extended the unit tests and ran: `make check` `make whitebox_crash_test` Reviewed By: ajkr Differential Revision: D21822714 Pulled By: ltamasi fbshipit-source-id: e2b29c8b6da1bf0f59004acc889e4870b2d18215		5 years ago
.circleci	pin image version in circle CI builds (#6876 )	5 years ago
.github/workflows	Clean up some code related to file checksums (#6861 )	5 years ago
buckifier	Allow missing "unversioned" python, as in CentOS 8 (#6883 )	5 years ago
build_tools	Allow missing "unversioned" python, as in CentOS 8 (#6883 )	5 years ago
cache	Update googletest from 1.8.1 to 1.10.0 (#6808 )	5 years ago
cmake	Add `find_dependency()` in cmake config file. (#6791 )	5 years ago
coverage	Find the correct gcov (#6904 )	5 years ago
db	Improve consistency checks in VersionBuilder (#6901 )	5 years ago
db_stress_tool	Fix potential overflow of unsigned type in for loop (#6902 )	5 years ago
docs	Log warning for high bits/key in legacy Bloom filter (#6312 )	5 years ago
env	Fix potential overflow of unsigned type in for loop (#6902 )	5 years ago
examples	Add missing my_pid to fprintf in multi_process_example (#6731 )	5 years ago
file	Replace Status with IOStatus in CopyFile and CreateFile (#6916 )	5 years ago
hdfs	prototype status check enforcement (#6798 )	5 years ago
include/rocksdb	Expose rocksdb_options_copy function to the C API (#6880 )	5 years ago
java	Add newer WBWI::NewIteratorWithBase functions to RocksJava (#6872 )	5 years ago
logging	Fix info log source file display length (#5824 )	5 years ago
memory	C++20 compatibility (#6697 )	5 years ago
memtable	C++20 compatibility (#6697 )	5 years ago
monitoring	Fix FilterBench when RTTI=0 (#6732 )	5 years ago
options	Allow MultiGet users to limit cumulative value size (#6826 )	5 years ago
port	C++20 compatibility (#6697 )	5 years ago
table	Fix handling of too-small filter partition size (#6905 )	5 years ago
test_util	Reduce dependency on gtest dependency in release code (#6907 )	5 years ago
third-party	Update googletest from 1.8.1 to 1.10.0 (#6808 )	5 years ago
tools	For ApproximateSizes, pro-rate table metadata size over data blocks (#6784 )	5 years ago
trace_replay	Fix multiple CF replay failure in db_bench replay (#6787 )	5 years ago
util	Fix potential overflow of unsigned type in for loop (#6902 )	5 years ago
utilities	Fix potential overflow of unsigned type in for loop (#6902 )	5 years ago
.clang-format	A script that automatically reformat affected lines	11 years ago
.gitignore	Allow missing "unversioned" python, as in CentOS 8 (#6883 )	5 years ago
.lgtm.yml	Create lgtm.yml for LGTM.com C/C++ analysis (#4058 )	7 years ago
.travis.yml	Misc things for ASSERT_STATUS_CHECKED, also gcc 4.8.5 (#6871 )	5 years ago
.watchmanconfig	Added .watchmanconfig file to rocksdb repo (#5593 )	6 years ago
AUTHORS	Update RocksDB Authors File	7 years ago
CMakeLists.txt	Update googletest from 1.8.1 to 1.10.0 (#6808 )	5 years ago
CODE_OF_CONDUCT.md	Adopt Contributor Covenant	5 years ago
CONTRIBUTING.md	Add Code of Conduct	7 years ago
COPYING	Add GPLv2 as an alternative license.	8 years ago
DEFAULT_OPTIONS_HISTORY.md	options.delayed_write_rate use the rate of rate_limiter by default.	8 years ago
DUMP_FORMAT.md	First version of rocksdb_dump and rocksdb_undump.	10 years ago
HISTORY.md	For ApproximateSizes, pro-rate table metadata size over data blocks (#6784 )	5 years ago
INSTALL.md	Update the version of the dependencies used by the RocksJava static build (#4761 )	6 years ago
LANGUAGE-BINDINGS.md	LANGUAGE-BINDINGS.md: mention python-rocksdb	6 years ago
LICENSE.Apache	Change RocksDB License	8 years ago
LICENSE.leveldb	Add back the LevelDB license file	8 years ago
Makefile	Update googletest from 1.8.1 to 1.10.0 (#6808 )	5 years ago
README.md	Add Slack forum to README (#6773 )	5 years ago
ROCKSDB_LITE.md	Fix some typos in comments and docs.	7 years ago
TARGETS	Update googletest from 1.8.1 to 1.10.0 (#6808 )	5 years ago
USERS.md	Add YugabyteDB to USERS (#6786 )	5 years ago
Vagrantfile	Adding CentOS 7 Vagrantfile & build script	7 years ago
WINDOWS_PORT.md	#5145 , rename port/dirent.h to port/port_dirent.h to avoid compile err when use port dir as header dir output (#5152 )	6 years ago
appveyor.yml	C++20 compatibility (#6697 )	5 years ago
defs.bzl	Make testpilot recognize that these tests have coverage instrumentation	5 years ago
issue_template.md	Add Google Group to Issue Template	5 years ago
src.mk	Update googletest from 1.8.1 to 1.10.0 (#6808 )	5 years ago
thirdparty.inc	Fix build jemalloc api (#5470 )	6 years ago

README.md

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/master/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.

Design discussions are conducted in https://www.facebook.com/groups/rocksdb.dev/ and https://rocksdb.slack.com/

License

RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.