rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

Ari Ekmekji 40c64434d4 Parallelize L0-L1 Compaction: Restructure Compaction Job Summary: As of now compactions involving files from Level 0 and Level 1 are single threaded because the files in L0, although sorted, are not range partitioned like the other levels. This means that during L0-L1 compaction each file from L1 needs to be merged with potentially all the files from L0. This attempt to parallelize the L0-L1 compaction assigns a thread and a corresponding iterator to each L1 file that then considers only the key range found in that L1 file and only the L0 files that have those keys (and only the specific portion of those L0 files in which those keys are found). In this way the overlap is minimized and potentially eliminated between different iterators focusing on the same files. The first step is to restructure the compaction logic to break L0-L1 compactions into multiple, smaller, sequential compactions. Eventually each of these smaller jobs will be run simultaneously. Areas to pay extra attention to are # Correct aggregation of compaction job statistics across multiple threads # Proper opening/closing of output files (make sure each thread's is unique) # Keys that span multiple L1 files # Skewed distributions of keys within L0 files Test Plan: Make and run db_test (newer version has separate compaction tests) and compaction_job_stats_test Reviewers: igor, noetzli, anthony, sdong, yhchiang Reviewed By: yhchiang Subscribers: MarkCallaghan, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42699		9 years ago
arcanist_util	Integrate Jenkins with Phabricator	10 years ago
build_tools	Commit both PR and internal code review changes	9 years ago
coverage	Fix coverage script	10 years ago
db	Parallelize L0-L1 Compaction: Restructure Compaction Job	9 years ago
doc	Remove seek compaction	11 years ago
examples	[API Change] Improve EventListener::OnFlushCompleted interface	10 years ago
hdfs	Improved FileExists API	9 years ago
include/rocksdb	Parallelize L0-L1 Compaction: Restructure Compaction Job	9 years ago
java	Add missing hashCode() implementation	9 years ago
port	Fix WinEnv::NowMicrosec	9 years ago
table	Improved FileExists API	9 years ago
third-party	"make format" against last 10 commits	9 years ago
tools	Fixed typos in db_stress	9 years ago
util	Parallelize L0-L1 Compaction: Restructure Compaction Job	9 years ago
utilities	WriteBatch Save Points	9 years ago
.arcconfig	Integrate Jenkins with Phabricator	10 years ago
.clang-format	A script that automatically reformat affected lines	11 years ago
.gitignore	Windows Port from Microsoft	10 years ago
.travis.yml	Don't preinstall jemalloc in Travis	10 years ago
AUTHORS	Add AUTHORS file. Fix #203	10 years ago
CMakeLists.txt	Add missing tests, fix db_sanity	9 years ago
CONTRIBUTING.md	facebook accounts are not required for CLA signers	11 years ago
DUMP_FORMAT.md	First version of rocksdb_dump and rocksdb_undump.	10 years ago
HISTORY.md	WriteBatch Save Points	9 years ago
INSTALL.md	Commit both PR and internal code review changes	9 years ago
LICENSE	Fix copyright year	11 years ago
Makefile	Add missing hashCode() implementation	9 years ago
PATENTS	Update Patent Grant.	10 years ago
README.md	Replaced "built on on earlier work" by "built on earlier work" in README.md	10 years ago
ROCKSDB_LITE.md	Optimistic Transactions	10 years ago
USERS.md	Add Yahoo's blog post about Sherpa to USERS.md	10 years ago
Vagrantfile	RocksDB on FreeBSD support	10 years ago
WINDOWS_PORT.md	Commit both PR and internal code review changes	9 years ago
src.mk	Move general compaction tests from db_test.cc to db_compaction_test.cc	9 years ago
thirdparty.inc	Conditional use of third-party libraries	9 years ago

README.md

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

This code is a library that forms the core building block for a fast key value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it specially suitable for storing multiple terabytes of data in a single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/master/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.

Design discussions are conducted in https://www.facebook.com/groups/rocksdb.dev/