rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

Andrew Kryczka 62f70f6d14 Reduce scope of compression dictionary to single SST (#4952 ) Summary: Our previous approach was to train one compression dictionary per compaction, using the first output SST to train a dictionary, and then applying it on subsequent SSTs in the same compaction. While this was great for minimizing CPU/memory/I/O overhead, it did not achieve good compression ratios in practice. In our most promising potential use case, moderate reductions in a dictionary's scope make a major difference on compression ratio. So, this PR changes compression dictionary to be scoped per-SST. It accepts the tradeoff during table building to use more memory and CPU. Important changes include: - The `BlockBasedTableBuilder` has a new state when dictionary compression is in-use: `kBuffered`. In that state it accumulates uncompressed data in-memory whenever `Add` is called. - After accumulating target file size bytes or calling `BlockBasedTableBuilder::Finish`, a `BlockBasedTableBuilder` moves to the `kUnbuffered` state. The transition (`EnterUnbuffered()`) involves sampling the buffered data, training a dictionary, and compressing/writing out all buffered data. In the `kUnbuffered` state, a `BlockBasedTableBuilder` behaves the same as before -- blocks are compressed/written out as soon as they fill up. - Samples are now whole uncompressed data blocks, except the final sample may be a partial data block so we don't breach the user's configured `max_dict_bytes` or `zstd_max_train_bytes`. The dictionary trainer is supposed to work better when we pass it real units of compression. Previously we were passing 64-byte KV samples which was not realistic. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4952 Differential Revision: D13967980 Pulled By: ajkr fbshipit-source-id: 82bea6f7537e1529c7a1a4cdee84585f5949300f		6 years ago
buckifier	Fix skylark incompatible build files in rocksdb	6 years ago
build_tools	Add latest toolchain (gcc-8, etc.) build support for fbcode users (#4923 )	6 years ago
cache	Revert "Move MemoryAllocator option from Cache to BlockBasedTableOpti… (#4697 )	6 years ago
cmake	Make FindZLIB consistent with official definitions (#4823 )	6 years ago
coverage	Remove unused imports, from python scripts. (#4057 )	6 years ago
db	Reduce scope of compression dictionary to single SST (#4952 )	6 years ago
docs	Insane line length detected (#4813 )	6 years ago
env	fix for nvme device path (#4866 )	6 years ago
examples	Pin top-level index on partitioned index/filter blocks (#4037 )	6 years ago
hdfs	Update all unique/shared_ptr instances to be qualified with namespace std (#4638 )	6 years ago
include/rocksdb	WritePrepared: add private options to TransactionDBOptions (#4966 )	6 years ago
java	Remove PlainTable's feature store_index_in_file (#4914 )	6 years ago
memtable	Remove cuckoo hash memtable (#4953 )	6 years ago
monitoring	Allow copy for PerfContext objects (#4919 )	6 years ago
options	Remove cuckoo hash memtable (#4953 )	6 years ago
port	Detect if Jemalloc is linked with the binary (#4844 )	6 years ago
table	Reduce scope of compression dictionary to single SST (#4952 )	6 years ago
third-party	Support pragma once in all header files and cleanup some warnings (#4339 )	6 years ago
tools	Reduce scope of compression dictionary to single SST (#4952 )	6 years ago
util	Reduce scope of compression dictionary to single SST (#4952 )	6 years ago
utilities	Enhance transaction_test_util with delays (#4970 )	6 years ago
.clang-format	A script that automatically reformat affected lines	11 years ago
.gitignore	RocksDB Trace Analyzer (#4091 )	6 years ago
.lgtm.yml	Create lgtm.yml for LGTM.com C/C++ analysis (#4058 )	6 years ago
.travis.yml	Fix printf formatting on MacOS (#4533 )	6 years ago
AUTHORS	Update RocksDB Authors File	7 years ago
CMakeLists.txt	Remove cuckoo hash memtable (#4953 )	6 years ago
CODE_OF_CONDUCT.md	Add Code of Conduct	7 years ago
CONTRIBUTING.md	Add Code of Conduct	7 years ago
COPYING	Add GPLv2 as an alternative license.	8 years ago
DEFAULT_OPTIONS_HISTORY.md	options.delayed_write_rate use the rate of rate_limiter by default.	8 years ago
DUMP_FORMAT.md	First version of rocksdb_dump and rocksdb_undump.	10 years ago
HISTORY.md	Reduce scope of compression dictionary to single SST (#4952 )	6 years ago
INSTALL.md	Update the version of the dependencies used by the RocksJava static build (#4761 )	6 years ago
LANGUAGE-BINDINGS.md	Added PingCaps Rust RocksDB and ObjectiveRocks (#4065 )	6 years ago
LICENSE.Apache	Change RocksDB License	7 years ago
LICENSE.leveldb	Add back the LevelDB license file	7 years ago
Makefile	Change the command to invoke parallel tests (#4922 )	6 years ago
README.md	Create lgtm.yml for LGTM.com C/C++ analysis (#4058 )	6 years ago
ROCKSDB_LITE.md	Fix some typos in comments and docs.	7 years ago
TARGETS	Remove cuckoo hash memtable (#4953 )	6 years ago
USERS.md	Adding IOTA Foundation to USERS.MD (#4436 )	6 years ago
Vagrantfile	Adding CentOS 7 Vagrantfile & build script	7 years ago
WINDOWS_PORT.md	Add GCC 8 to Travis (#3433 )	6 years ago
appveyor.yml	Add RocksJava build to AppVeyor	6 years ago
issue_template.md	Add a template for issues	7 years ago
src.mk	Remove cuckoo hash memtable (#4953 )	6 years ago
thirdparty.inc	Provide a way to override windows memory allocator with jemalloc for ZSTD	7 years ago

README.md

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

This code is a library that forms the core building block for a fast key value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it specially suitable for storing multiple terabytes of data in a single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/master/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.

Design discussions are conducted in https://www.facebook.com/groups/rocksdb.dev/

License

RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.