rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

Yanqin Jin 1cda273dc3 Fix a silent data loss for write-committed txn (#9571 ) Summary: The following sequence of events can cause silent data loss for write-committed transactions. ``` Time thread 1 bg flush \| db->Put("a") \| txn = NewTxn() \| txn->Put("b", "v") \| txn->Prepare() // writes only to 5.log \| db->SwitchMemtable() // memtable 1 has "a" \| // close 5.log, \| // creates 8.log \| trigger flush \| pick memtable 1 \| unlock db mutex \| write new sst \| txn->ctwb->Put("gtid", "1") // writes 8.log \| txn->Commit() // writes to 8.log \| // writes to memtable 2 \| compute min_log_number_to_keep_2pc, this \| will be 8 (incorrect). \| \| Purge obsolete wals, including 5.log \| V ``` At this point, writes of txn exists only in memtable. Close db without flush because db thinks the data in memtable are backed by log. Then reopen, the writes are lost except key-value pair {"gtid"->"1"}, only the commit marker of txn is in 8.log The reason lies in `PrecomputeMinLogNumberToKeep2PC()` which calls `FindMinPrepLogReferencedByMemTable()`. In the above example, when bg flush thread tries to find obsolete wals, it uses the information computed by `PrecomputeMinLogNumberToKeep2PC()`. The return value of `PrecomputeMinLogNumberToKeep2PC()` depends on three components - `PrecomputeMinLogNumberToKeepNon2PC()`. This represents the WAL that has unflushed data. As the name of this method suggests, it does not account for 2PC. Although the keys reside in the prepare section of a previous WAL, the column family references the current WAL when they are actually inserted into the memtable during txn commit. - `prep_tracker->FindMinLogContainingOutstandingPrep()`. This represents the WAL with a prepare section but the txn hasn't committed. - `FindMinPrepLogReferencedByMemTable()`. This represents the WAL on which some memtables (mutable and immutable) depend for their unflushed data. The bug lies in `FindMinPrepLogReferencedByMemTable()`. Originally, this function skips checking the column families that are being flushed, but the unit test added in this PR shows that they should not be. In this unit test, there is only the default column family, and one of its memtables has unflushed data backed by a prepare section in 5.log. We should return this information via `FindMinPrepLogReferencedByMemTable()`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9571 Test Plan: ``` ./transaction_test --gtest_filter=/TransactionTest.SwitchMemtableDuringPrepareAndCommit_WC/ make check ``` Reviewed By: siying Differential Revision: D34235236 Pulled By: riversand963 fbshipit-source-id: 120eb21a666728a38dda77b96276c6af72b008b1		3 years ago
.circleci	Remove cat_ignore_eagain (#9531 )	3 years ago
.github/workflows	Add (& fix) some simple source code checks (#8821 )	3 years ago
buckifier	Use -Wno-invalid-offsetof instead of dangerous offset_of hack (#9563 )	3 years ago
build_tools	Fix parallel test updates in CI; fbcode LIB_MODE=shared (#9553 )	3 years ago
cache	Require C++17 (#9481 )	3 years ago
cmake	gcc-11 and cmake related cleanup (#9286 )	3 years ago
coverage	Remove asan_symbolize.py for internal asan build (#8737 )	3 years ago
db	Fix a silent data loss for write-committed txn (#9571 )	3 years ago
db_stress_tool	Remove timestamp from key in expected state (#9525 )	3 years ago
docs	New blog post for Ribbon filter (#8992 )	3 years ago
env	Return different Status based on ObjectRegistry::NewObject calls (#9333 )	3 years ago
examples	Require C++17 (#9481 )	3 years ago
file	Disallow a combination of options (#9348 )	3 years ago
fuzz	Fix compilation errors and add fuzzers to CircleCI (#9420 )	3 years ago
include/rocksdb	Remove deprecated RemoteCompaction API (#9570 )	3 years ago
java	Update Java API for FilterPolicy changes (#9569 )	3 years ago
logging	Use system-wide thread ID in info log lines (#9164 )	3 years ago
memory	Return different Status based on ObjectRegistry::NewObject calls (#9333 )	3 years ago
memtable	Remove using namespace (#9369 )	3 years ago
microbench	Refactor FilterPolicies toward Customizable (#9567 )	3 years ago
monitoring	Restore Regex support for ObjectLibrary::Register, rename new APIs to allow old one to be deprecated in the future (#9362 )	3 years ago
options	Refactor FilterPolicies toward Customizable (#9567 )	3 years ago
plugin	Add initial CMake support to plugin (#9214 )	3 years ago
port	Require C++17 (#9481 )	3 years ago
table	Fix assertion failure in FastLocalBloomBitsBuilder (#9585 )	3 years ago
test_util	Use the comparator from the sst file table properties in sst_dump_tool (#9491 )	3 years ago
third-party	Work around some new clang-analyze failures (#9515 )	3 years ago
tools	LDB to add --secondary_path to help (#9582 )	3 years ago
trace_replay	Added `TraceOptions::preserve_write_order` (#9334 )	3 years ago
util	Refactor FilterPolicies toward Customizable (#9567 )	3 years ago
utilities	Fix a silent data loss for write-committed txn (#9571 )	3 years ago
.clang-format	A script that automatically reformat affected lines	11 years ago
.gitignore	gitignore cmake-build-* for CLion integration (#7933 )	4 years ago
.lgtm.yml	Create lgtm.yml for LGTM.com C/C++ analysis (#4058 )	6 years ago
.travis.yml	Temporary disable Travis s390x Makefile build (#9512 )	3 years ago
.watchmanconfig	Added .watchmanconfig file to rocksdb repo (#5593 )	5 years ago
AUTHORS	Update RocksDB Authors File	7 years ago
CMakeLists.txt	Use -Wno-invalid-offsetof instead of dangerous offset_of hack (#9563 )	3 years ago
CODE_OF_CONDUCT.md	Adopt Contributor Covenant	5 years ago
CONTRIBUTING.md	Add Code of Conduct	7 years ago
COPYING	Add GPLv2 as an alternative license.	8 years ago
DEFAULT_OPTIONS_HISTORY.md	Add Options::DisableExtraChecks, clarify force_consistency_checks (#9363 )	3 years ago
DUMP_FORMAT.md	First version of rocksdb_dump and rocksdb_undump.	10 years ago
HISTORY.md	Fix a silent data loss for write-committed txn (#9571 )	3 years ago
INSTALL.md	Clarify Google benchmark < 1.6.0 in INSTALL.md (#9505 )	3 years ago
LANGUAGE-BINDINGS.md	Update branch name to "main" in README/LANGUAGE_BINDINGS (#8727 )	3 years ago
LICENSE.Apache	Change RocksDB License	7 years ago
LICENSE.leveldb	Add back the LevelDB license file	7 years ago
Makefile	Use -Wno-invalid-offsetof instead of dangerous offset_of hack (#9563 )	3 years ago
PLUGINS.md	Move RADOS support to separate repo (#9206 )	3 years ago
README.md	README: De-list slack channel, list Google group (#9387 )	3 years ago
ROCKSDB_LITE.md	Fix some typos in comments and docs.	7 years ago
TARGETS	Use -Wno-invalid-offsetof instead of dangerous offset_of hack (#9563 )	3 years ago
USERS.md	Add Solana's RocksDB use case in USERS.md (#9558 )	3 years ago
Vagrantfile	Adding CentOS 7 Vagrantfile & build script	7 years ago
WINDOWS_PORT.md	Update branch name in WINDOWS_PORT.md (#8745 )	3 years ago
defs.bzl	Make testpilot recognize that these tests have coverage instrumentation	5 years ago
issue_template.md	Add Google Group to Issue Template	5 years ago
src.mk	Introduce a CountedFileSystem for counting file operations (#9283 )	3 years ago
thirdparty.inc	Fix build jemalloc api (#5470 )	5 years ago

README.md

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/main/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.

Questions and discussions are welcome on the RocksDB Developers Public Facebook group and email list on Google Groups.

License

RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.