rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

Peter Dillinger ec11c23caa Add thread safety to BackupEngine, explain more (#8115 ) Summary: BackupEngine previously had unclear but strict concurrency requirements that the API user must follow for safe use. Now we make that clear, by separating operations into "Read," "Append," and "Write" operations, and specifying which combinations are safe across threads on the same BackupEngine object (previously none; now all, using a read-write lock), and which are safe across different BackupEngine instances open on the same backup_dir. The changes to backupable_db.h should be backward compatible. It is mostly about eliminating copies of what should be the same function and (unsurprisingly) useful documentation comments were often placed on only one of the two copies. With the re-organization, we are also grouping different categories of operations. In the future we might add BackupEngineReadAppendOnly, but that didn't seem necessary. To mark API Read operations 'const', I had to mark some implementation functions 'const' and some fields mutable. Functional changes: * Added RWMutex locking around public API functions to implement thread safety on a single object. To avoid future bugs, this is another internal class layered on top (removing many "override" in BackupEngineImpl). It would be possible to allow more concurrency between operations, rather than mutual exclusion, but IMHO not worth the work. * Fixed a race between Open() (Initialize()) and CreateNewBackup() for different objects on the same backup_dir, where Initialize() could delete the temporary meta file created during CreateNewBackup(). (This was found by the new test.) Also cleaned up a couple of "status checked" TODOs, and improved a checksum mismatch error message to include involved files. Potential follow-up work: * CreateNewBackup has an API wart because it doesn't tell you the BackupID it just created, which makes it of limited use in a multithreaded setting. * We could also consider a Refresh() function to catch up to changes made from another BackupEngine object to the same dir. * Use a lock file to prevent multiple writer BackupEngines, but this won't work on remote filesystems not supporting lock files. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8115 Test Plan: new mini-stress test in backup unit tests, run with gcc, clang, ASC, TSAN, and UBSAN, 100 iterations each. Reviewed By: ajkr Differential Revision: D27347589 Pulled By: pdillinger fbshipit-source-id: 28d82ed2ac672e44085a739ddb19d297dad14b15		4 years ago
.circleci	Add circleci format_compatible nightly build (#7926 )	4 years ago
.github/workflows	Update clang-format-diff.py path (#7944 )	4 years ago
buckifier	range_tree requires GNU libc on ppc64 (#8070 )	4 years ago
build_tools	range_tree requires GNU libc on ppc64 (#8070 )	4 years ago
cache	Fix some typos in comments (#8066 )	4 years ago
cmake	Add `find_dependency()` in cmake config file. (#6791 )	5 years ago
coverage	Find the correct gcov (#6904 )	5 years ago
db	Compaction should not move data to up level (#8116 )	4 years ago
db_stress_tool	Add user-defined timestamps to db_stress (#8061 )	4 years ago
docs	Update github-pages and dependencies (#7850 )	4 years ago
env	Adding safer permissions to PosixFilesystem::NewLogger (#8106 )	4 years ago
examples	make:Fix c header prototypes (#7994 )	4 years ago
file	fix typo (#8118 )	4 years ago
fuzz	Remove Legacy and Custom FileWrapper classes from header files (#7851 )	4 years ago
hdfs	fix build with 'USE_HDFS' on windows (#6950 )	5 years ago
include/rocksdb	Add thread safety to BackupEngine, explain more (#8115 )	4 years ago
java	Fix possible mistype in a comment (#8086 )	4 years ago
logging	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033 )	4 years ago
memory	Use thread-safe `strerror_r()` to get error message (#8087 )	4 years ago
memtable	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033 )	4 years ago
monitoring	Add the statistics and info log for Error handler (#8050 )	4 years ago
options	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033 )	4 years ago
plugin	Makefile support to statically link external plugin code (#7918 )	4 years ago
port	On ARM platform, use yield op to relax CPU. See issue 7376 (#7438 )	4 years ago
table	Apply `sample_for_compression` to all block-based tables (#8105 )	4 years ago
test_util	Add ROCKSDB_GTEST_BYPASS (#8048 )	4 years ago
third-party	Fix a compilation error in CircleCI vs2019 CXX20 (#8090 )	4 years ago
tools	Fix check in db_bench for num shard bits to match check in LRUCache (#8110 )	4 years ago
trace_replay	Fix clang analyze for trace replace GetPayload (#8121 )	4 years ago
util	Introduce a ThreadGuard class and use it in ExternalSSTFileTest.PickedLevelBug (#8112 )	4 years ago
utilities	Add thread safety to BackupEngine, explain more (#8115 )	4 years ago
.clang-format	A script that automatically reformat affected lines	11 years ago
.gitignore	gitignore cmake-build-* for CLion integration (#7933 )	4 years ago
.lgtm.yml	Create lgtm.yml for LGTM.com C/C++ analysis (#4058 )	7 years ago
.travis.yml	use `LIB_MODE=shared` on Travis `make` commands (#8043 )	4 years ago
.watchmanconfig	Added .watchmanconfig file to rocksdb repo (#5593 )	6 years ago
AUTHORS	Update RocksDB Authors File	7 years ago
CMakeLists.txt	Move compacted_db_impl.[c\|h] to db/db_impl (#8082 )	4 years ago
CODE_OF_CONDUCT.md	Adopt Contributor Covenant	5 years ago
CONTRIBUTING.md	Add Code of Conduct	7 years ago
COPYING	Add GPLv2 as an alternative license.	8 years ago
DEFAULT_OPTIONS_HISTORY.md	options.delayed_write_rate use the rate of rate_limiter by default.	8 years ago
DUMP_FORMAT.md	First version of rocksdb_dump and rocksdb_undump.	10 years ago
HISTORY.md	Add thread safety to BackupEngine, explain more (#8115 )	4 years ago
INSTALL.md	Update the version of the dependencies used by the RocksJava static build (#4761 )	6 years ago
LANGUAGE-BINDINGS.md	Add RestoreDBFromLatestBackup to C API, add new C# package (#7092 )	5 years ago
LICENSE.Apache	Change RocksDB License	8 years ago
LICENSE.leveldb	Add back the LevelDB license file	8 years ago
Makefile	range_tree requires GNU libc on ppc64 (#8070 )	4 years ago
PLUGINS.md	Makefile support to statically link external plugin code (#7918 )	4 years ago
README.md	Fix the CI badge for ppc64le Jenkins (#7561 )	4 years ago
ROCKSDB_LITE.md	Fix some typos in comments and docs.	7 years ago
TARGETS	Move compacted_db_impl.[c\|h] to db/db_impl (#8082 )	4 years ago
USERS.md	Add Apache Doris to USERS (#7865 )	4 years ago
Vagrantfile	Adding CentOS 7 Vagrantfile & build script	7 years ago
WINDOWS_PORT.md	#5145 , rename port/dirent.h to port/port_dirent.h to avoid compile err when use port dir as header dir output (#5152 )	6 years ago
appveyor.yml	Remove 2019 from appveyor (#7038 )	5 years ago
defs.bzl	Make testpilot recognize that these tests have coverage instrumentation	5 years ago
issue_template.md	Add Google Group to Issue Template	5 years ago
src.mk	range_tree requires GNU libc on ppc64 (#8070 )	4 years ago
thirdparty.inc	Fix build jemalloc api (#5470 )	6 years ago

README.md

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/master/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.

Design discussions are conducted in https://www.facebook.com/groups/rocksdb.dev/ and https://rocksdb.slack.com/

License

RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.