rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

Peter Dillinger 6ac1d25fd0 Fix+clean up handling of mock sleeps (#7101 ) Summary: We have a number of tests hanging on MacOS and windows due to mishandling of code for mock sleeps. In addition, the code was in terrible shape because the same variable (addon_time_) would sometimes refer to microseconds and sometimes to seconds. One test even assumed it was nanoseconds but was written to pass anyway. This has been cleaned up so that DB tests generally use a SpecialEnv function to mock sleep, for either some number of microseconds or seconds depending on the function called. But to call one of these, the test must first call SetMockSleep (precondition enforced with assertion), which also turns sleeps in RocksDB into mock sleeps. To also removes accounting for actual clock time, call SetTimeElapseOnlySleepOnReopen, which implies SetMockSleep (on DB re-open). This latter setting only works by applying on DB re-open, otherwise havoc can ensue if Env goes back in time with DB open. More specifics: Removed some unused test classes, and updated comments on the general problem. Fixed DBSSTTest.GetTotalSstFilesSize using a sync point callback instead of mock time. For this we have the only modification to production code, inserting a sync point callback in flush_job.cc, which is not a change to production behavior. Removed unnecessary resetting of mock times to 0 in many tests. RocksDB deals in relative time. Any behaviors relying on absolute date/time are likely a bug. (The above test DBSSTTest.GetTotalSstFilesSize was the only one clearly injecting a specific absolute time for actual testing convenience.) Just in case I misunderstood some test, I put this note in each replacement: // NOTE: Presumed unnecessary and removed: resetting mock time in env Strengthened some tests like MergeTestTime, MergeCompactionTimeTest, and FilterCompactionTimeTest in db_test.cc stats_history_test and blob_db_test are each their own beast, rather deeply dependent on MockTimeEnv. Each gets its own variant of a work-around for TimedWait in a mock time environment. (Reduces redundancy and inconsistency in stats_history_test.) Intended follow-up: Remove TimedWait from the public API of InstrumentedCondVar, and only make that accessible through Env by passing in an InstrumentedCondVar and a deadline. Then the Env implementations mocking time can fix this problem without using sync points. (Test infrastructure using sync points interferes with individual tests' control over sync points.) With that change, we can simplify/consolidate the scattered work-arounds. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7101 Test Plan: make check on Linux and MacOS Reviewed By: zhichao-cao Differential Revision: D23032815 Pulled By: pdillinger fbshipit-source-id: 7f33967ada8b83011fb54e8279365c008bd6610b		4 years ago
.circleci	CircleCI: apt retries for downloads from unreliable llvm.org (#7234 )	4 years ago
.github/workflows	Clean up some code related to file checksums (#6861 )	5 years ago
buckifier	Exclude c_test from buck build opt mode (#7093 )	4 years ago
build_tools	Makefile support for link-time optimization (#7181 )	4 years ago
cache	Add more tests to ASSERT_STATUS_CHECKED (#7211 )	4 years ago
cmake	Add `find_dependency()` in cmake config file. (#6791 )	5 years ago
coverage	Find the correct gcov (#6904 )	5 years ago
db	Fix+clean up handling of mock sleeps (#7101 )	4 years ago
db_stress_tool	Mark files for compaction in stress/crash tests (#7231 )	4 years ago
docs	remediation of S205607	4 years ago
env	Remove duplicate colon in Status message (#7041 )	4 years ago
examples	add WITH_EXAMPLES options to cmake and cleanups. (#6580 )	4 years ago
file	Enable IO timeouts for iterators (#7161 )	4 years ago
hdfs	fix build with 'USE_HDFS' on windows (#6950 )	4 years ago
include/rocksdb	Enable IO timeouts for iterators (#7161 )	4 years ago
java	Automatically number the Maven artifacts (#7219 )	4 years ago
logging	Fix info log source file display length (#5824 )	5 years ago
memory	C++20 compatibility (#6697 )	5 years ago
memtable	More Makefile Cleanup (#7097 )	4 years ago
monitoring	Fix+clean up handling of mock sleeps (#7101 )	4 years ago
options	Enable IO timeouts for iterators (#7161 )	4 years ago
port	build fixes for GNU/kFreeBSD (#6992 )	4 years ago
table	Fix cmake build on MacOS (#7205 )	4 years ago
test_util	Fix+clean up handling of mock sleeps (#7101 )	4 years ago
third-party	Revert "Update googletest from 1.8.1 to 1.10.0 (#6808 )" (#6923 )	5 years ago
tools	Mark files for compaction in stress/crash tests (#7231 )	4 years ago
trace_replay	Add support to start and end IOTracing through DB APIs (#7203 )	4 years ago
util	Fix+clean up handling of mock sleeps (#7101 )	4 years ago
utilities	Fix+clean up handling of mock sleeps (#7101 )	4 years ago
.clang-format	A script that automatically reformat affected lines	11 years ago
.gitignore	Allow missing "unversioned" python, as in CentOS 8 (#6883 )	5 years ago
.lgtm.yml	Create lgtm.yml for LGTM.com C/C++ analysis (#4058 )	6 years ago
.travis.yml	More Makefile clean-up (#7066 )	4 years ago
.watchmanconfig	Added .watchmanconfig file to rocksdb repo (#5593 )	5 years ago
AUTHORS	Update RocksDB Authors File	7 years ago
CMakeLists.txt	Fix cmake build on MacOS (#7205 )	4 years ago
CODE_OF_CONDUCT.md	Adopt Contributor Covenant	5 years ago
CONTRIBUTING.md	Add Code of Conduct	7 years ago
COPYING	Add GPLv2 as an alternative license.	8 years ago
DEFAULT_OPTIONS_HISTORY.md	options.delayed_write_rate use the rate of rate_limiter by default.	8 years ago
DUMP_FORMAT.md	First version of rocksdb_dump and rocksdb_undump.	10 years ago
HISTORY.md	Fix a perf regression that caused every key to go through upper bound check (#7209 )	4 years ago
INSTALL.md	Update the version of the dependencies used by the RocksJava static build (#4761 )	6 years ago
LANGUAGE-BINDINGS.md	Add RestoreDBFromLatestBackup to C API, add new C# package (#7092 )	4 years ago
LICENSE.Apache	Change RocksDB License	7 years ago
LICENSE.leveldb	Add back the LevelDB license file	7 years ago
Makefile	Allow optimization level to be set in Makefile (#7202 )	4 years ago
README.md	Add CircleCI gadget (#7028 )	4 years ago
ROCKSDB_LITE.md	Fix some typos in comments and docs.	7 years ago
TARGETS	Replace tracked_keys with a new LockTracker interface in TransactionDB (#7013 )	4 years ago
USERS.md	Add YugabyteDB to USERS (#6786 )	5 years ago
Vagrantfile	Adding CentOS 7 Vagrantfile & build script	7 years ago
WINDOWS_PORT.md	#5145 , rename port/dirent.h to port/port_dirent.h to avoid compile err when use port dir as header dir output (#5152 )	6 years ago
appveyor.yml	Remove 2019 from appveyor (#7038 )	4 years ago
defs.bzl	Make testpilot recognize that these tests have coverage instrumentation	5 years ago
issue_template.md	Add Google Group to Issue Template	5 years ago
src.mk	Replace tracked_keys with a new LockTracker interface in TransactionDB (#7013 )	4 years ago
thirdparty.inc	Fix build jemalloc api (#5470 )	5 years ago

README.md

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/master/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.

Design discussions are conducted in https://www.facebook.com/groups/rocksdb.dev/ and https://rocksdb.slack.com/

License

RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.