rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

Andrew Kryczka 843d2e3137 Shared dictionary compression using reference block Summary: This adds a new metablock containing a shared dictionary that is used to compress all data blocks in the SST file. The size of the shared dictionary is configurable in CompressionOptions and defaults to 0. It's currently only used for zlib/lz4/lz4hc, but the block will be stored in the SST regardless of the compression type if the user chooses a nonzero dictionary size. During compaction, computes the dictionary by randomly sampling the first output file in each subcompaction. It pre-computes the intervals to sample by assuming the output file will have the maximum allowable length. In case the file is smaller, some of the pre-computed sampling intervals can be beyond end-of-file, in which case we skip over those samples and the dictionary will be a bit smaller. After the dictionary is generated using the first file in a subcompaction, it is loaded into the compression library before writing each block in each subsequent file of that subcompaction. On the read path, gets the dictionary from the metablock, if it exists. Then, loads that dictionary into the compression library before reading each block. Test Plan: new unit test Reviewers: yhchiang, IslamAbdelRahman, cyan, sdong Reviewed By: sdong Subscribers: andrewkr, yoshinorim, kradhakrishnan, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D52287		9 years ago
arcanist_util	Propagate sandcastle run error to UI	9 years ago
build_tools	Print memory allocation counters	9 years ago
coverage	Fix coverage script	10 years ago
db	Shared dictionary compression using reference block	9 years ago
doc	Lint everything	9 years ago
examples	Adding pin_l0_filter_and_index_blocks_in_cache feature and related fixes.	9 years ago
hdfs	using java7 in runtime for hdfs env (#1072 )	9 years ago
include/rocksdb	Shared dictionary compression using reference block	9 years ago
java	Delete deprecated backup classes in Java	9 years ago
memtable	Updated all copyright headers to the new format.	9 years ago
port	Alpine Linux Build (#990 )	9 years ago
table	Shared dictionary compression using reference block	9 years ago
third-party	Fix the build break on Ubuntu 15.10 when gcc 5.2.1 is used	9 years ago
tools	Shared dictionary compression using reference block	9 years ago
util	Shared dictionary compression using reference block	9 years ago
utilities	Fix BackupableDBTest	9 years ago
.arcconfig	Integrate Jenkins with Phabricator	10 years ago
.clang-format	A script that automatically reformat affected lines	11 years ago
.gitignore	Ignore db_test2	9 years ago
.travis.yml	Split Travis unittests Job	9 years ago
AUTHORS	Add AUTHORS file. Fix #203	10 years ago
CMakeLists.txt	Introduce XPRESS compresssion on Windows. (#1081 )	9 years ago
CONTRIBUTING.md	facebook accounts are not required for CLA signers	11 years ago
DEFAULT_OPTIONS_HISTORY.md	Release RocksDB 4.7	9 years ago
DUMP_FORMAT.md	First version of rocksdb_dump and rocksdb_undump.	10 years ago
HISTORY.md	Shared dictionary compression using reference block	9 years ago
INSTALL.md	Simple changes to support builds for ppc64[le] consistent with X86	9 years ago
LANGUAGE-BINDINGS.md	Merge pull request #1056 from facebook/igorcanadi-patch-1	9 years ago
LICENSE	Updated all copyright headers to the new format.	9 years ago
Makefile	Print memory allocation counters	9 years ago
PATENTS	Update Patent Grant.	10 years ago
README.md	Replaced "built on on earlier work" by "built on earlier work" in README.md	10 years ago
ROCKSDB_LITE.md	Optimistic Transactions	10 years ago
USERS.md	Update USERS.md with link to LinkedIn blog post (#1088 )	9 years ago
Vagrantfile	RocksDB on FreeBSD support	10 years ago
WINDOWS_PORT.md	Commit both PR and internal code review changes	10 years ago
appveyor.yml	Exclude DBTest.FileCreationRandomFailure as a long running test	9 years ago
src.mk	Split db_test.cc	9 years ago
thirdparty.inc	Introduce XPRESS compresssion on Windows. (#1081 )	9 years ago

README.md

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

This code is a library that forms the core building block for a fast key value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it specially suitable for storing multiple terabytes of data in a single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/master/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.

Design discussions are conducted in https://www.facebook.com/groups/rocksdb.dev/