rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

Peter Dillinger fc9d4071f0 Fast path for detecting unchanged prefix_extractor (#9407 ) Summary: Fixes a major performance regression in 6.26, where extra CPU is spent in SliceTransform::AsString when reads involve a prefix_extractor (Get, MultiGet, Seek). Common case performance is now better than 6.25. This change creates a "fast path" for verifying that the current prefix extractor is unchanged and compatible with what was used to generate a table file. This fast path detects the common case by pointer comparison on the current prefix_extractor and a "known good" prefix extractor (if applicable) that is saved at the time the table reader is opened. The "known good" prefix extractor is saved as another shared_ptr copy (in an existing field, however) to ensure the pointer is not recycled. When the prefix_extractor has changed to a different instance but same compatible configuration (rare, odd), performance is still a regression compared to 6.25, but this is likely acceptable because of the oddity of such a case. The performance of incompatible prefix_extractor is essentially unchanged. Also fixed a minor case (ForwardIterator) where a prefix_extractor could be used via a raw pointer after being freed as a shared_ptr, if replaced via SetOptions. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9407 Test Plan: ## Performance Populate DB with `TEST_TMPDIR=/dev/shm/rocksdb ./db_bench -benchmarks=fillrandom -num=10000000 -disable_wal=1 -write_buffer_size=10000000 -bloom_bits=16 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=0 -prefix_size=12` Running head-to-head comparisons simultaneously with `TEST_TMPDIR=/dev/shm/rocksdb ./db_bench -use_existing_db -readonly -benchmarks=seekrandom -num=10000000 -duration=20 -disable_wal=1 -bloom_bits=16 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=0 -prefix_size=12` Below each is compared by ops/sec vs. baseline which is version 6.25 (multiple baseline runs because of variable machine load) v6.26: 4833 vs. 6698 (<- major regression!) v6.27: 4737 vs. 6397 (still) New: 6704 vs. 6461 (better than baseline in common case) Disabled fastpath: 4843 vs. 6389 (e.g. if prefix extractor instance changes but is still compatible) Changed prefix size (no usable filter) in new: 787 vs. 5927 Changed prefix size (no usable filter) in new & baseline: 773 vs. 784 Reviewed By: mrambacher Differential Revision: D33677812 Pulled By: pdillinger fbshipit-source-id: 571d9711c461fb97f957378a061b7e7dbc4d6a76		2 years ago
.circleci	Remove pyenv installation and use deps from S3 (#9406 )	2 years ago
.github/workflows	Add (& fix) some simple source code checks (#8821 )	3 years ago
buckifier	Update TARGETS and related scripts (#9310 )	3 years ago
build_tools	Use fcntl(F_FULLFSYNC) on OS X (#9356 )	2 years ago
cache	Fix unity build with SUPPORT_CLOCK_CACHE (#9309 )	3 years ago
cmake	gcc-11 and cmake related cleanup (#9286 )	3 years ago
coverage	Remove asan_symbolize.py for internal asan build (#8737 )	3 years ago
db	Fast path for detecting unchanged prefix_extractor (#9407 )	2 years ago
db_stress_tool	Test correctness with WAL disabled in non-txn blackbox crash tests (#9338 )	3 years ago
docs	New blog post for Ribbon filter (#8992 )	3 years ago
env	Use fcntl(F_FULLFSYNC) on OS X (#9356 )	2 years ago
examples	Remove using namespace (#9369 )	2 years ago
file	Fix a bug causing duplicate trailing entries in WritableFile (buffered IO) (#9236 )	3 years ago
fuzz	Remove using namespace (#9369 )	2 years ago
hdfs	Make the Env class Customizable (#9293 )	3 years ago
include/rocksdb	Mark destructors as override (#9404 )	2 years ago
java	Add support for Apple Silicon to RocksJava (#9254 )	2 years ago
logging	Use system-wide thread ID in info log lines (#9164 )	3 years ago
memory	Fix compilation error when building static_lib (#9377 )	2 years ago
memtable	Remove using namespace (#9369 )	2 years ago
microbench	Skip directory fsync for filesystem btrfs (#8903 )	3 years ago
monitoring	Restore Regex support for ObjectLibrary::Register, rename new APIs to allow old one to be deprecated in the future (#9362 )	2 years ago
options	Add Options::DisableExtraChecks, clarify force_consistency_checks (#9363 )	2 years ago
plugin	Add initial CMake support to plugin (#9214 )	3 years ago
port	Remove using namespace (#9369 )	2 years ago
table	Fast path for detecting unchanged prefix_extractor (#9407 )	2 years ago
test_util	Restore Regex support for ObjectLibrary::Register, rename new APIs to allow old one to be deprecated in the future (#9362 )	2 years ago
third-party	Remove using namespace (#9369 )	2 years ago
tools	Add Options::DisableExtraChecks, clarify force_consistency_checks (#9363 )	2 years ago
trace_replay	Added `TraceOptions::preserve_write_order` (#9334 )	3 years ago
util	Fast path for detecting unchanged prefix_extractor (#9407 )	2 years ago
utilities	Range Locking: add support for escalation barriers (#9290 )	2 years ago
.clang-format	A script that automatically reformat affected lines	11 years ago
.gitignore	gitignore cmake-build-* for CLion integration (#7933 )	3 years ago
.lgtm.yml	Create lgtm.yml for LGTM.com C/C++ analysis (#4058 )	6 years ago
.travis.yml	Re-enable 390x+cmake* Travis jobs (#9110 )	3 years ago
.watchmanconfig	Added .watchmanconfig file to rocksdb repo (#5593 )	5 years ago
AUTHORS	Update RocksDB Authors File	7 years ago
CMakeLists.txt	Use fcntl(F_FULLFSYNC) on OS X (#9356 )	2 years ago
CODE_OF_CONDUCT.md	Adopt Contributor Covenant	5 years ago
CONTRIBUTING.md	Add Code of Conduct	7 years ago
COPYING	Add GPLv2 as an alternative license.	7 years ago
DEFAULT_OPTIONS_HISTORY.md	Add Options::DisableExtraChecks, clarify force_consistency_checks (#9363 )	2 years ago
DUMP_FORMAT.md	First version of rocksdb_dump and rocksdb_undump.	9 years ago
HISTORY.md	Fast path for detecting unchanged prefix_extractor (#9407 )	2 years ago
INSTALL.md	Update installation instructions (#8158 )	3 years ago
LANGUAGE-BINDINGS.md	Update branch name to "main" in README/LANGUAGE_BINDINGS (#8727 )	3 years ago
LICENSE.Apache	Change RocksDB License	7 years ago
LICENSE.leveldb	Add back the LevelDB license file	7 years ago
Makefile	Add support for Apple Silicon to RocksJava (#9254 )	2 years ago
PLUGINS.md	Add ZenFS to plugin list (#8218 )	3 years ago
README.md	README: De-list slack channel, list Google group (#9387 )	2 years ago
ROCKSDB_LITE.md	Fix some typos in comments and docs.	6 years ago
TARGETS	Update TARGETS and related scripts (#9310 )	3 years ago
USERS.md	Update USERS.md (#8923 )	3 years ago
Vagrantfile	Adding CentOS 7 Vagrantfile & build script	6 years ago
WINDOWS_PORT.md	Update branch name in WINDOWS_PORT.md (#8745 )	3 years ago
appveyor.yml	Remove 2019 from appveyor (#7038 )	4 years ago
defs.bzl	Make testpilot recognize that these tests have coverage instrumentation	4 years ago
issue_template.md	Add Google Group to Issue Template	4 years ago
src.mk	Make MemoryAllocator into a Customizable class (#8980 )	3 years ago
thirdparty.inc	Fix build jemalloc api (#5470 )	5 years ago

README.md

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/main/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.

Questions and discussions are welcome on the RocksDB Developers Public Facebook group and email list on Google Groups.

License

RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.