rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

Peter Dillinger 2a383f21f4 Add Bloom/Ribbon hybrid API support (#8679 ) Summary: This is essentially resurrection and fixing of the part of https://github.com/facebook/rocksdb/issues/8198 that was reverted in https://github.com/facebook/rocksdb/issues/8212, using data added in https://github.com/facebook/rocksdb/issues/8246. Basically, when configuring Ribbon filter, you can specify an LSM level before which Bloom will be used instead of Ribbon. But Bloom is only considered for Leveled and Universal compaction styles and file going into a known LSM level. This way, SST file writer, FIFO compaction, etc. use Ribbon filter as you would expect with NewRibbonFilterPolicy. So that this can be controlled with a single int value and so that flushes can be distinguished from intra-L0, we consider flush to go to level -1 for the purposes of this option. (Explained in API comment.) I also expect the most common and recommended Ribbon configuration to use Bloom during flush, to minimize slowing down writes and because according to my estimates, Ribbon only pays off if the structure lives in memory for more than an hour. Thus, I have changed the default for NewRibbonFilterPolicy to be this mild hybrid configuration. I don't really want to add something like NewHybridFilterPolicy because at least the mild hybrid configuration (Bloom for flush, Ribbon otherwise) should be considered a natural choice. C APIs also updated, but because they don't support overloading, rocksdb_filterpolicy_create_ribbon is kept pure ribbon for clarity and rocksdb_filterpolicy_create_ribbon_hybrid must be called for a hybrid configuration. While touching C API, I changed bits per key options from int to double. BuiltinFilterPolicy is needed so that LevelThresholdFilterPolicy doesn't inherit unused fields from BloomFilterPolicy. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8679 Test Plan: new + updated tests, including crash test Reviewed By: jay-zhuang Differential Revision: D30445797 Pulled By: pdillinger fbshipit-source-id: 6f5aeddfd6d79f7e55493b563c2d1d2d568892e1		3 years ago
.circleci	Add missing steps for cmake build (#8524 )	3 years ago
.github/workflows	Update clang-format-diff.py path (#7944 )	4 years ago
buckifier	Modify script which generates TARGETS (#8366 )	3 years ago
build_tools	Pass extra db_stress args to fbcode crash tests (#8587 )	3 years ago
cache	Add a PerfContext counter for secondary cache hits (#8685 )	3 years ago
cmake	Add `find_dependency()` in cmake config file. (#6791 )	5 years ago
coverage	Find the correct gcov (#6904 )	4 years ago
db	Add Bloom/Ribbon hybrid API support (#8679 )	3 years ago
db_stress_tool	Add Bloom/Ribbon hybrid API support (#8679 )	3 years ago
docs	Bump addressable from 2.7.0 to 2.8.0 in /docs (#8515 )	3 years ago
env	Make TraceRecord and Replayer public (#8611 )	3 years ago
examples	make:Fix c header prototypes (#7994 )	4 years ago
file	Add db_test2 to to ASSERT_STATUS_CHECKED (#8640 )	3 years ago
fuzz	Make EventListener into a Customizable Class (#8473 )	3 years ago
hdfs	fix build with 'USE_HDFS' on windows (#6950 )	4 years ago
include/rocksdb	Add Bloom/Ribbon hybrid API support (#8679 )	3 years ago
java	Add a stat to count secondary cache hits (#8666 )	3 years ago
logging	Do not attempt to rename non-existent info log (#8622 )	3 years ago
memory	Use thread-safe `strerror_r()` to get error message (#8087 )	4 years ago
memtable	Improve MemPurge sampling (#8656 )	3 years ago
microbench	Add micro-benchmark support (#8493 )	3 years ago
monitoring	Add a PerfContext counter for secondary cache hits (#8685 )	3 years ago
options	Add Bloom/Ribbon hybrid API support (#8679 )	3 years ago
plugin	Makefile support to statically link external plugin code (#7918 )	4 years ago
port	Support dynamic sector size in alignment validation for Windows. (#8613 )	3 years ago
table	Add Bloom/Ribbon hybrid API support (#8679 )	3 years ago
test_util	Fix some minor issues in the Customizable infrastructure (#8566 )	3 years ago
third-party	Fix a compilation error in CircleCI vs2019 CXX20 (#8090 )	4 years ago
tools	Add Bloom/Ribbon hybrid API support (#8679 )	3 years ago
trace_replay	Add `IteratorTraceExecutionResult` for iterator related trace records. (#8687 )	3 years ago
util	Add Bloom/Ribbon hybrid API support (#8679 )	3 years ago
utilities	Allow Replayer to report the results of TraceRecords. (#8657 )	3 years ago
.clang-format	A script that automatically reformat affected lines	11 years ago
.gitignore	gitignore cmake-build-* for CLion integration (#7933 )	4 years ago
.lgtm.yml	Create lgtm.yml for LGTM.com C/C++ analysis (#4058 )	6 years ago
.travis.yml	Move arm build from travis to circleci (#8203 )	4 years ago
.watchmanconfig	Added .watchmanconfig file to rocksdb repo (#5593 )	5 years ago
AUTHORS	Update RocksDB Authors File	7 years ago
CMakeLists.txt	Allow Replayer to report the results of TraceRecords. (#8657 )	3 years ago
CODE_OF_CONDUCT.md	Adopt Contributor Covenant	5 years ago
CONTRIBUTING.md	Add Code of Conduct	7 years ago
COPYING	Add GPLv2 as an alternative license.	8 years ago
DEFAULT_OPTIONS_HISTORY.md	options.delayed_write_rate use the rate of rate_limiter by default.	8 years ago
DUMP_FORMAT.md	First version of rocksdb_dump and rocksdb_undump.	10 years ago
HISTORY.md	Add Bloom/Ribbon hybrid API support (#8679 )	3 years ago
INSTALL.md	Update installation instructions (#8158 )	4 years ago
LANGUAGE-BINDINGS.md	Add RestoreDBFromLatestBackup to C API, add new C# package (#7092 )	4 years ago
LICENSE.Apache	Change RocksDB License	7 years ago
LICENSE.leveldb	Add back the LevelDB license file	7 years ago
Makefile	Add db_test2 to to ASSERT_STATUS_CHECKED (#8640 )	3 years ago
PLUGINS.md	Add ZenFS to plugin list (#8218 )	4 years ago
README.md	Fix the CI badge for ppc64le Jenkins (#7561 )	4 years ago
ROCKSDB_LITE.md	Fix some typos in comments and docs.	7 years ago
TARGETS	Allow Replayer to report the results of TraceRecords. (#8657 )	3 years ago
USERS.md	Add Apache Doris to USERS (#7865 )	4 years ago
Vagrantfile	Adding CentOS 7 Vagrantfile & build script	7 years ago
WINDOWS_PORT.md	#5145 , rename port/dirent.h to port/port_dirent.h to avoid compile err when use port dir as header dir output (#5152 )	6 years ago
appveyor.yml	Remove 2019 from appveyor (#7038 )	4 years ago
defs.bzl	Make testpilot recognize that these tests have coverage instrumentation	5 years ago
issue_template.md	Add Google Group to Issue Template	5 years ago
src.mk	Allow Replayer to report the results of TraceRecords. (#8657 )	3 years ago
thirdparty.inc	Fix build jemalloc api (#5470 )	5 years ago

README.md

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/master/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.

Design discussions are conducted in https://www.facebook.com/groups/rocksdb.dev/ and https://rocksdb.slack.com/

License

RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.