rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

Sagar Vemuri 1cd45cd1b3 FIFO Compaction with TTL Summary: Introducing FIFO compactions with TTL. FIFO compaction is based on size only which makes it tricky to enable in production as use cases can have organic growth. A user requested an option to drop files based on the time of their creation instead of the total size. To address that request: - Added a new TTL option to FIFO compaction options. - Updated FIFO compaction score to take TTL into consideration. - Added a new table property, creation_time, to keep track of when the SST file is created. - Creation_time is set as below: - On Flush: Set to the time of flush. - On Compaction: Set to the max creation_time of all the files involved in the compaction. - On Repair and Recovery: Set to the time of repair/recovery. - Old files created prior to this code change will have a creation_time of 0. - FIFO compaction with TTL is enabled when ttl > 0. All files older than ttl will be deleted during compaction. i.e. `if (file.creation_time < (current_time - ttl)) then delete(file)`. This will enable cases where you might want to delete all files older than, say, 1 day. - FIFO compaction will fall back to the prior way of deleting files based on size if: - the creation_time of all files involved in compaction is 0. - the total size (of all SST files combined) does not drop below `compaction_options_fifo.max_table_files_size` even if the files older than ttl are deleted. This feature is not supported if max_open_files != -1 or with table formats other than Block-based. Test Plan: Added tests. Benchmark results: Base: FIFO with max size: 100MB :: ``` svemuri@dev15905 ~/rocksdb (fifo-compaction) $ TEST_TMPDIR=/dev/shm ./db_bench --benchmarks=readwhilewriting --num=5000000 --threads=16 --compaction_style=2 --fifo_compaction_max_table_files_size_mb=100 readwhilewriting : 1.924 micros/op 519858 ops/sec; 13.6 MB/s (1176277 of 5000000 found) ``` With TTL (a low one for testing) :: ``` svemuri@dev15905 ~/rocksdb (fifo-compaction) $ TEST_TMPDIR=/dev/shm ./db_bench --benchmarks=readwhilewriting --num=5000000 --threads=16 --compaction_style=2 --fifo_compaction_max_table_files_size_mb=100 --fifo_compaction_ttl=20 readwhilewriting : 1.902 micros/op 525817 ops/sec; 13.7 MB/s (1185057 of 5000000 found) ``` Example Log lines: ``` 2017/06/26-15:17:24.609249 7fd5a45ff700 (Original Log Time 2017/06/26-15:17:24.609177) [db/compaction_picker.cc:1471] [default] FIFO compaction: picking file 40 with creation time 1498515423 for deletion 2017/06/26-15:17:24.609255 7fd5a45ff700 (Original Log Time 2017/06/26-15:17:24.609234) [db/db_impl_compaction_flush.cc:1541] [default] Deleted 1 files ... 2017/06/26-15:17:25.553185 7fd5a61a5800 [DEBUG] [db/db_impl_files.cc:309] [JOB 0] Delete /dev/shm/dbbench/000040.sst type=2 #40 -- OK 2017/06/26-15:17:25.553205 7fd5a61a5800 EVENT_LOG_v1 {"time_micros": 1498515445553199, "job": 0, "event": "table_file_deletion", "file_number": 40} ``` SST Files remaining in the dbbench dir, after db_bench execution completed: ``` svemuri@dev15905 ~/rocksdb (fifo-compaction) $ ls -l /dev/shm//dbbench/*.sst -rw-r--r--. 1 svemuri users 30749887 Jun 26 15:17 /dev/shm//dbbench/000042.sst -rw-r--r--. 1 svemuri users 30768779 Jun 26 15:17 /dev/shm//dbbench/000044.sst -rw-r--r--. 1 svemuri users 30757481 Jun 26 15:17 /dev/shm//dbbench/000046.sst ``` Closes https://github.com/facebook/rocksdb/pull/2480 Differential Revision: D5305116 Pulled By: sagar0 fbshipit-source-id: 3e5cfcf5dd07ed2211b5b37492eb235b45139174		7 years ago
arcanist_util	Fix arc setting for Facebook internal tools	8 years ago
buckifier	Fix TARGETS file tests list	7 years ago
build_tools	fixed typo	7 years ago
cache	Add GPLv2 as an alternative license.	7 years ago
cmake/modules	CMake: more MinGW fixes	7 years ago
coverage	Fix coverage script	10 years ago
db	FIFO Compaction with TTL	7 years ago
docs	Intra-L0 blog post	7 years ago
env	Encryption at rest support	7 years ago
examples	CMake: more MinGW fixes	7 years ago
hdfs	New API for background work in single thread pool	7 years ago
include/rocksdb	FIFO Compaction with TTL	7 years ago
java	CLANG Tidy	7 years ago
memtable	WriteBufferManager will not trigger flush if much data is already being flushed	7 years ago
monitoring	revert perf_context and io_stats to __thread	7 years ago
options	FIFO Compaction with TTL	7 years ago
port	Implement ReopenWritibaleFile on Windows and other fixes	7 years ago
table	FIFO Compaction with TTL	7 years ago
third-party	fixed typo	7 years ago
tools	FIFO Compaction with TTL	7 years ago
util	revert perf_context and io_stats to __thread	7 years ago
utilities	Fix TARGETS file tests list	7 years ago
.clang-format	A script that automatically reformat affected lines	11 years ago
.deprecated_arcconfig	Update ShipIt to honor TARGETS updates	7 years ago
.gitignore	Simple blob file dumper	7 years ago
.travis.yml	Force travis to build with clang on MacOS	7 years ago
AUTHORS	Add AUTHORS file. Fix #203	10 years ago
CMakeLists.txt	Fix TARGETS file tests list	7 years ago
CONTRIBUTING.md	facebook accounts are not required for CLA signers	10 years ago
COPYING	Add GPLv2 as an alternative license.	7 years ago
DEFAULT_OPTIONS_HISTORY.md	options.delayed_write_rate use the rate of rate_limiter by default.	7 years ago
DUMP_FORMAT.md	First version of rocksdb_dump and rocksdb_undump.	9 years ago
HISTORY.md	FIFO Compaction with TTL	7 years ago
INSTALL.md	cross-platform compatibility improvements	7 years ago
LANGUAGE-BINDINGS.md	Adding Dlang to the list	7 years ago
LICENSE	Updated all copyright headers to the new format.	9 years ago
Makefile	Fix TARGETS file tests list	7 years ago
PATENTS	Update Patent Grant.	9 years ago
README.md	Appveyor badge to show master branch	8 years ago
ROCKSDB_LITE.md	Optimistic Transactions	9 years ago
TARGETS	Fix TARGETS file tests list	7 years ago
USERS.md	fixed typo	7 years ago
Vagrantfile	Update Vagrant file (test internal phabricator workflow)	8 years ago
WINDOWS_PORT.md	Commit both PR and internal code review changes	9 years ago
appveyor.yml	Rework test running script.	7 years ago
src.mk	Fix TARGETS file tests list	7 years ago
thirdparty.inc	Introduce XPRESS compresssion on Windows. (#1081 )	8 years ago

README.md

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

This code is a library that forms the core building block for a fast key value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it specially suitable for storing multiple terabytes of data in a single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/master/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.

Design discussions are conducted in https://www.facebook.com/groups/rocksdb.dev/