rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

Yu Zhang 15053f3ab4 Logically strip timestamp during flush (#11557 ) Summary: Logically strip the user-defined timestamp when L0 files are created during flush when `AdvancedColumnFamilyOptions.persist_user_defined_timestamps` is false. Logically stripping timestamp here means replacing the original user-defined timestamp with a mininum timestamp, which for now is hard coded to be all zeros bytes. While working on this, I caught a missing piece on the `BlockBuilder` level for this feature. The current quick path `std::min(buffer_size, last_key_size)` needs a bit tweaking to work for this feature. When user-defined timestamp is stripped during block building, on writing first entry or right after resetting, `buffer` is empty and `buffer_size` is zero as usual. However, in follow-up writes, depending on the size of the stripped user-defined timestamp, and the size of the value, what's in `buffer` can sometimes be smaller than `last_key_size`, leading `std::min(buffer_size, last_key_size)` to truncate the `last_key`. Previous test doesn't caught the bug because in those tests, the size of the stripped user-defined timestamps bytes is smaller than the length of the value. In order to avoid the conditional operation, this PR changed the original trivial `std::min` operation into an arithmetic operation. Since this is a change in a hot and performance critical path, I did the following benchmark to check no observable regression is introduced. ```TEST_TMPDIR=/dev/shm/rocksdb1 ./db_bench -benchmarks=fillseq -memtablerep=vector -allow_concurrent_memtable_write=false -num=50000000``` Compiled with DEBUG_LEVEL=0 Test vs. control runs simulaneous for better accuracy, units = ops/sec PR vs base: Round 1: 350652 vs 349055 Round 2: 365733 vs 364308 Round 3: 355681 vs 354475 Pull Request resolved: https://github.com/facebook/rocksdb/pull/11557 Test Plan: New timestamp specific test added or existing tests augmented, both are parameterized with `UserDefinedTimestampTestMode`: `UserDefinedTimestampTestMode::kNormal` -> UDT feature enabled, write / read with min timestamp `UserDefinedTimestampTestMode::kStripUserDefinedTimestamps` -> UDT feature enabled, write / read with min timestamp, set Options.persist_user_defined_timestamps to false. ``` make all check ./db_wal_test --gtest_filter="WithTimestamp" ./flush_job_test --gtest_filter="WithTimestamp" ./repair_test --gtest_filter="WithTimestamp" ./block_based_table_reader_test ``` Reviewed By: pdillinger Differential Revision: D47027664 Pulled By: jowlyzhang fbshipit-source-id: e729193b6334dfc63aaa736d684d907a022571f5		2 years ago
.circleci	Revert enabling IO uring in db_stress (#11242 )	2 years ago
.github/workflows	Fix failed CI job "Check buck targets and code format" (#11532 )	2 years ago
buckifier	Fix duplicate symbols in linking with buck2 (#11421 )	2 years ago
build_tools	Repair/instate jemalloc build on M1 (#11257 )	2 years ago
cache	Update secondary_cache_adapter.cc (#11566 )	2 years ago
cmake	gcc-11 and cmake related cleanup (#9286 )	3 years ago
coverage	Remove platform009 and default to platform010 (#11333 )	2 years ago
db	Logically strip timestamp during flush (#11557 )	2 years ago
db_stress_tool	Initialize StressTest::optimistic_txn_db_ in ctor (#11547 )	2 years ago
docs	Fix typo in twitter link (#11529 )	2 years ago
env	remove duplicate comments in EncryptedEnv (#11549 )	2 years ago
examples	Remove RocksDB LITE (#11147 )	2 years ago
file	Fix extra prefetching when num_file_reads_for_auto_readahead is 1 in async_io (#11560 )	2 years ago
fuzz	Block per key-value checksum (#11287 )	2 years ago
include/rocksdb	Logically strip timestamp during flush (#11557 )	2 years ago
java	Make option `level_compaction_dynamic_level_bytes` true by default (#11525 )	2 years ago
logging	Disabling some IO error assertion in EnvLogger (#11314 )	2 years ago
memory	Change internal headers with duplicate names (#11408 )	2 years ago
memtable	remove redundant move (#11418 )	2 years ago
microbench	Small improvements to DBGet microbenchmark (#11498 )	2 years ago
monitoring	Add a ticker to track number of trash files deleted in background thread (#11540 )	2 years ago
options	Change internal headers with duplicate names (#11408 )	2 years ago
plugin	Add initial CMake support to plugin (#9214 )	3 years ago
port	Add OpenBSD Support (#11255 )	2 years ago
table	Logically strip timestamp during flush (#11557 )	2 years ago
test_util	Add support to strip / pad timestamp when writing / reading a block (#11472 )	2 years ago
third-party	fix optimization-disabled test builds with platform010 (#11361 )	2 years ago
tools	Add skip_tmpdir_check arg in crash script (#11539 )	2 years ago
trace_replay	Fix error maybe-uninitialized #11100 (#11101 )	2 years ago
unreleased_history	Update HISTORY.md for branch cut for 8.4.fb (#11565 )	2 years ago
util	Add OpenBSD Support (#11255 )	2 years ago
utilities	Use FaultInjectionTestFS in transaction_test, clarify Close() APIs (#11499 )	2 years ago
.clang-format	A script that automatically reformat affected lines	11 years ago
.gitignore	Tweak on IsTrivialMove() (#11467 )	2 years ago
.lgtm.yml	Create lgtm.yml for LGTM.com C/C++ analysis (#4058 )	7 years ago
.watchmanconfig	Added .watchmanconfig file to rocksdb repo (#5593 )	6 years ago
AUTHORS	Update RocksDB Authors File	7 years ago
CMakeLists.txt	Add utils to use for handling user defined timestamp size record in WAL (#11451 )	2 years ago
CODE_OF_CONDUCT.md	Adopt Contributor Covenant	5 years ago
CONTRIBUTING.md	Add Code of Conduct	7 years ago
COPYING	Add GPLv2 as an alternative license.	8 years ago
DEFAULT_OPTIONS_HISTORY.md	Add Options::DisableExtraChecks, clarify force_consistency_checks (#9363 )	3 years ago
DUMP_FORMAT.md	First version of rocksdb_dump and rocksdb_undump.	10 years ago
HISTORY.md	Update HISTORY.md for branch cut for 8.4.fb (#11565 )	2 years ago
INSTALL.md	Simplify detection of x86 CPU features (#11419 )	2 years ago
LANGUAGE-BINDINGS.md	Add grocksdb in Go language bindings (#10498 )	2 years ago
LICENSE.Apache	Change RocksDB License	8 years ago
LICENSE.leveldb	Add back the LevelDB license file	8 years ago
Makefile	Add utils to use for handling user defined timestamp size record in WAL (#11451 )	2 years ago
PLUGINS.md	Added encryption plugin based on Intel open-source ipp-crypto library (#11429 )	2 years ago
README.md	Remove deprecated integration tests from README.md (#11354 )	2 years ago
TARGETS	Add utils to use for handling user defined timestamp size record in WAL (#11451 )	2 years ago
USERS.md	Add more users (#11369 )	2 years ago
Vagrantfile	Adding CentOS 7 Vagrantfile & build script	7 years ago
WINDOWS_PORT.md	Update branch name in WINDOWS_PORT.md (#8745 )	3 years ago
common.mk	Clean up variables for temporary directory (#9961 )	3 years ago
crash_test.mk	Stress/Crash Test for OptimisticTransactionDB (#11513 )	2 years ago
issue_template.md	Add Google Group to Issue Template	5 years ago
rocksdb.pc.in	build: fix pkg-config file generation (#9953 )	3 years ago
src.mk	Add utils to use for handling user defined timestamp size record in WAL (#11451 )	2 years ago
thirdparty.inc	Fix build jemalloc api (#5470 )	6 years ago

README.md

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/main/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.

Questions and discussions are welcome on the RocksDB Developers Public Facebook group and email list on Google Groups.

License

RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.