rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

Peter Dillinger 57f3032285 Allow fractional bits/key in BloomFilterPolicy (#6092 ) Summary: There's no technological impediment to allowing the Bloom filter bits/key to be non-integer (fractional/decimal) values, and it provides finer control over the memory vs. accuracy trade-off. This is especially handy in using the format_version=5 Bloom filter in place of the old one, because bits_per_key=9.55 provides the same accuracy as the old bits_per_key=10. This change not only requires refining the logic for choosing the best num_probes for a given bits/key setting, it revealed a flaw in that logic. As bits/key gets higher, the best num_probes for a cache-local Bloom filter is closer to bpk / 2 than to bpk * 0.69, the best choice for a standard Bloom filter. For example, at 16 bits per key, the best num_probes is 9 (FP rate = 0.0843%) not 11 (FP rate = 0.0884%). This change fixes and refines that logic (for the format_version=5 Bloom filter only, just in case) based on empirical tests to find accuracy inflection points between each num_probes. Although bits_per_key is now specified as a double, the new Bloom filter converts/rounds this to "millibits / key" for predictable/precise internal computations. Just in case of unforeseen compatibility issues, we round to the nearest whole number bits / key for the legacy Bloom filter, so as not to unlock new behaviors for it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6092 Test Plan: unit tests included Differential Revision: D18711313 Pulled By: pdillinger fbshipit-source-id: 1aa73295f152a995328cb846ef9157ae8a05522a		5 years ago
buckifier	Abandon use of folly::Optional (#6036 )	5 years ago
build_tools	New Bloom filter implementation for full and partitioned filters (#6007 )	5 years ago
cache	Misc hashing updates / upgrades (#5909 )	5 years ago
cmake	Compatible changes for cmake (#6045 )	5 years ago
coverage	Fix interpreter lines for files with python2-only syntax.	6 years ago
db	Make default value of options.ttl to be 30 days when it is supported. (#6073 )	5 years ago
docs	Blog post for write_unprepared (#5711 )	5 years ago
env	Add Env::SanitizeEnvOptions (#5885 )	5 years ago
examples	Add asserts in transaction example (#6055 )	5 years ago
file	Apply formatter to recent 200+ commits. (#5830 )	5 years ago
hdfs	Add copyright headers per FB open-source checkup tool. (#5199 )	6 years ago
include/rocksdb	Allow fractional bits/key in BloomFilterPolicy (#6092 )	5 years ago
java	Allow fractional bits/key in BloomFilterPolicy (#6092 )	5 years ago
logging	Apply formatter to recent 200+ commits. (#5830 )	5 years ago
memory	Charge block cache for cache internal usage (#5797 )	5 years ago
memtable	Misc hashing updates / upgrades (#5909 )	5 years ago
monitoring	Apply formatter to recent 200+ commits. (#5830 )	5 years ago
options	Allow fractional bits/key in BloomFilterPolicy (#6092 )	5 years ago
port	Fix block cache ID uniqueness for Windows builds (#5844 )	5 years ago
table	Allow fractional bits/key in BloomFilterPolicy (#6092 )	5 years ago
test_util	New Bloom filter implementation for full and partitioned filters (#6007 )	5 years ago
third-party	Refactor/consolidate legacy Bloom implementation details (#5784 )	5 years ago
tools	Update HISTORY.md for forward compatibility (#6085 )	5 years ago
trace_replay	Misc hashing updates / upgrades (#5909 )	5 years ago
util	Allow fractional bits/key in BloomFilterPolicy (#6092 )	5 years ago
utilities	Refactor blob file creation logic (#6066 )	5 years ago
.clang-format	A script that automatically reformat affected lines	11 years ago
.gitignore	Make buckifier python3 compatible (#5922 )	5 years ago
.lgtm.yml	Create lgtm.yml for LGTM.com C/C++ analysis (#4058 )	7 years ago
.travis.yml	Partial rebalance of TEST_GROUPs for Travis (#6010 )	5 years ago
.watchmanconfig	Added .watchmanconfig file to rocksdb repo (#5593 )	6 years ago
AUTHORS	Update RocksDB Authors File	7 years ago
CMakeLists.txt	Use lowercase for shlwapi.lib rpcrt4.lib (#6076 )	5 years ago
CODE_OF_CONDUCT.md	Adopt Contributor Covenant	5 years ago
CONTRIBUTING.md	Add Code of Conduct	7 years ago
COPYING	Add GPLv2 as an alternative license.	8 years ago
DEFAULT_OPTIONS_HISTORY.md	options.delayed_write_rate use the rate of rate_limiter by default.	8 years ago
DUMP_FORMAT.md	First version of rocksdb_dump and rocksdb_undump.	10 years ago
HISTORY.md	Allow fractional bits/key in BloomFilterPolicy (#6092 )	5 years ago
INSTALL.md	Update the version of the dependencies used by the RocksJava static build (#4761 )	6 years ago
LANGUAGE-BINDINGS.md	LANGUAGE-BINDINGS.md: mention python-rocksdb	6 years ago
LICENSE.Apache	Change RocksDB License	8 years ago
LICENSE.leveldb	Add back the LevelDB license file	8 years ago
Makefile	Fix naming of library on PPC64LE (#6080 )	5 years ago
README.md	Replaced some words (#5877 )	5 years ago
ROCKSDB_LITE.md	Fix some typos in comments and docs.	7 years ago
TARGETS	Abandon use of folly::Optional (#6036 )	5 years ago
USERS.md	Add avrio to USERS.md (#5748 )	5 years ago
Vagrantfile	Adding CentOS 7 Vagrantfile & build script	7 years ago
WINDOWS_PORT.md	#5145 , rename port/dirent.h to port/port_dirent.h to avoid compile err when use port dir as header dir output (#5152 )	6 years ago
appveyor.yml	New API to get all merge operands for a Key (#5604 )	5 years ago
defs.bzl	Add clarifying/instructive header to TARGETS and defs.bzl	5 years ago
issue_template.md	Add a template for issues	7 years ago
src.mk	FilterPolicy consolidation, part 2/2 (#5966 )	5 years ago
thirdparty.inc	Fix build jemalloc api (#5470 )	6 years ago

README.md

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/master/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.

Design discussions are conducted in https://www.facebook.com/groups/rocksdb.dev/

License

RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.