rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

Andrew Kryczka 6fbe96baf8 Compaction Support for Range Deletion Summary: This diff introduces RangeDelAggregator, which takes ownership of iterators provided to it via AddTombstones(). The tombstones are organized in a two-level map (snapshot stripe -> begin key -> tombstone). Tombstone creation avoids data copy by holding Slices returned by the iterator, which remain valid thanks to pinning. For compaction, we create a hierarchical range tombstone iterator with structure matching the iterator over compaction input data. An aggregator based on that iterator is used by CompactionIterator to determine which keys are covered by range tombstones. In case of merge operand, the same aggregator is used by MergeHelper. Upon finishing each file in the compaction, relevant range tombstones are added to the output file's range tombstone metablock and file boundaries are updated accordingly. To check whether a key is covered by range tombstone, RangeDelAggregator::ShouldDelete() considers tombstones in the key's snapshot stripe. When this function is used outside of compaction, it also checks newer stripes, which can contain covering tombstones. Currently the intra-stripe check involves a linear scan; however, in the future we plan to collapse ranges within a stripe such that binary search can be used. RangeDelAggregator::AddToBuilder() adds all range tombstones in the table's key-range to a new table's range tombstone meta-block. Since range tombstones may fall in the gap between files, we may need to extend some files' key-ranges. The strategy is (1) first file extends as far left as possible and other files do not extend left, (2) all files extend right until either the start of the next file or the end of the last range tombstone in the gap, whichever comes first. One other notable change is adding release/move semantics to ScopedArenaIterator such that it can be used to transfer ownership of an arena-allocated iterator, similar to how unique_ptr is used for malloc'd data. Depends on D61473 Test Plan: compaction_iterator_test, mock_table, end-to-end tests in D63927 Reviewers: sdong, IslamAbdelRahman, wanning, yhchiang, lightmark Reviewed By: lightmark Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D62205		8 years ago
arcanist_util	Fix arcanist	8 years ago
build_tools	Support ZSTD with finalized format	8 years ago
cmake/modules	cmake support for linux and osx (#1358 )	8 years ago
coverage	Fix coverage script	10 years ago
db	Compaction Support for Range Deletion	8 years ago
docs	Editorial change to README.md	8 years ago
examples	Fix typo (#903 )	8 years ago
hdfs	Fix bug in UnScSigned-off-by: xh931076284 <931076284@qq.com> (#1336 )	8 years ago
include/rocksdb	add seeforprev in history	8 years ago
java	Remove "-Xcheck:jni" from Java tests (#1402 )	8 years ago
memtable	Add SeekForPrev() to Iterator	8 years ago
port	Implement WinRandomRW file and improve code reuse (#1388 )	8 years ago
table	Compaction Support for Range Deletion	8 years ago
third-party	Fix clang analyzer errors	8 years ago
tools	Fix a minor bug in the ldb tool that was not selecting the specified (#1399 )	8 years ago
util	Remove function local statics that interfere with memory pooling (#1392 )	8 years ago
utilities	Make Lock Info test multiple column families	8 years ago
.arcconfig	Integrate Jenkins with Phabricator	10 years ago
.clang-format	A script that automatically reformat affected lines	11 years ago
.gitignore	Experiments on column-aware encodings	8 years ago
.travis.yml	Fix travis	8 years ago
AUTHORS	Add AUTHORS file. Fix #203	10 years ago
CMakeLists.txt	Compaction Support for Range Deletion	8 years ago
CONTRIBUTING.md	facebook accounts are not required for CLA signers	10 years ago
DEFAULT_OPTIONS_HISTORY.md	Release RocksDB 4.8.0	9 years ago
DUMP_FORMAT.md	First version of rocksdb_dump and rocksdb_undump.	9 years ago
HISTORY.md	add seeforprev in history	8 years ago
INSTALL.md	Simple changes to support builds for ppc64[le] consistent with X86	9 years ago
LANGUAGE-BINDINGS.md	Update LANGUAGE-BINDINGS.md	8 years ago
LICENSE	Updated all copyright headers to the new format.	9 years ago
Makefile	Minor fixes around Windows 64 Java Artifacts (#1366 )	8 years ago
PATENTS	Update Patent Grant.	10 years ago
README.md	Appveyor badge to show master branch	8 years ago
ROCKSDB_LITE.md	Optimistic Transactions	9 years ago
USERS.md	Adding Dgraph to list of Users (#1291 )	8 years ago
Vagrantfile	RocksDB on FreeBSD support	10 years ago
WINDOWS_PORT.md	Commit both PR and internal code review changes	9 years ago
appveyor.yml	fix vs generator (#1269 )	8 years ago
src.mk	Compaction Support for Range Deletion	8 years ago
thirdparty.inc	Introduce XPRESS compresssion on Windows. (#1081 )	9 years ago

README.md

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

This code is a library that forms the core building block for a fast key value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it specially suitable for storing multiple terabytes of data in a single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/master/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.

Design discussions are conducted in https://www.facebook.com/groups/rocksdb.dev/