rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

Zhichao Cao 699f45049d Introduce a mechanism to dump out blocks from block cache and re-insert to secondary cache (#8912 ) Summary: Background: Cache warming up will cause potential read performance degradation due to reading blocks from storage to the block cache. Since in production, the workload and access pattern to a certain DB is stable, it is a potential solution to dump out the blocks belonging to a certain DB to persist storage (e.g., to a file) and bulk-load the blocks to Secondary cache before the DB is relaunched. For example, when migrating a DB form host A to host B, it will take a short period of time, the access pattern to blocks in the block cache will not change much. It is efficient to dump out the blocks of certain DB, migrate to the destination host and insert them to the Secondary cache before we relaunch the DB. Design: we introduce the interface of CacheDumpWriter and CacheDumpRead for user to store the blocks dumped out from block cache. RocksDB will encode all the information and send the string to the writer. User can implement their own writer it they want. CacheDumper and CacheLoad are introduced to save the blocks and load the blocks respectively. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8912 Test Plan: add new tests to lru_cache_test and pass make check. Reviewed By: pdillinger Differential Revision: D31452871 Pulled By: zhichao-cao fbshipit-source-id: 11ab4f5d03e383f476947116361d54188d36ec48		3 years ago
.circleci	Fix and detect headers with missing dependencies (#8893 )	3 years ago
.github/workflows	Add (& fix) some simple source code checks (#8821 )	3 years ago
buckifier	Modify script which generates TARGETS (#8366 )	4 years ago
build_tools	Add microbench build support for fbcode (#8954 )	3 years ago
cache	Introduce a mechanism to dump out blocks from block cache and re-insert to secondary cache (#8912 )	3 years ago
cmake	Add `find_dependency()` in cmake config file. (#6791 )	5 years ago
coverage	Remove asan_symbolize.py for internal asan build (#8737 )	3 years ago
db	Misc doc fixes (#8983 )	3 years ago
db_stress_tool	Enable SingleDelete with user defined ts in db_bench and crash tests (#8971 )	3 years ago
docs	Misc doc fixes (#8983 )	3 years ago
env	Cleanup includes in dbformat.h (#8930 )	3 years ago
examples	Add (& fix) some simple source code checks (#8821 )	3 years ago
file	Introduce a mechanism to dump out blocks from block cache and re-insert to secondary cache (#8912 )	3 years ago
fuzz	Make EventListener into a Customizable Class (#8473 )	3 years ago
hdfs	fix build with 'USE_HDFS' on windows (#6950 )	5 years ago
include/rocksdb	Introduce a mechanism to dump out blocks from block cache and re-insert to secondary cache (#8912 )	3 years ago
java	Add remote compaction read/write bytes statistics (#8939 )	3 years ago
logging	Make SystemClock into a Customizable Class (#8636 )	3 years ago
memory	Fix and detect headers with missing dependencies (#8893 )	3 years ago
memtable	Make MemTableRepFactory into a Customizable class (#8419 )	3 years ago
microbench	Add micro-benchmark support (#8493 )	4 years ago
monitoring	Remove IOSTATS_ADD_IF_POSITIVE() (#8984 )	3 years ago
options	Cleanup includes in dbformat.h (#8930 )	3 years ago
plugin	Makefile support to statically link external plugin code (#7918 )	4 years ago
port	Prevent deadlock in db_stress with DbStressCompactionFilter (#8956 )	3 years ago
table	Introduce a mechanism to dump out blocks from block cache and re-insert to secondary cache (#8912 )	3 years ago
test_util	Cleanup multiple implementations of VectorIterator (#8901 )	3 years ago
third-party	Fix a compilation error in CircleCI vs2019 CXX20 (#8090 )	4 years ago
tools	Enable SingleDelete with user defined ts in db_bench and crash tests (#8971 )	3 years ago
trace_replay	Cleanup includes in dbformat.h (#8930 )	3 years ago
util	Cleanup multiple implementations of VectorIterator (#8901 )	3 years ago
utilities	Introduce a mechanism to dump out blocks from block cache and re-insert to secondary cache (#8912 )	3 years ago
.clang-format	A script that automatically reformat affected lines	11 years ago
.gitignore	gitignore cmake-build-* for CLion integration (#7933 )	4 years ago
.lgtm.yml	Create lgtm.yml for LGTM.com C/C++ analysis (#4058 )	7 years ago
.travis.yml	Move arm build from travis to circleci (#8203 )	4 years ago
.watchmanconfig	Added .watchmanconfig file to rocksdb repo (#5593 )	6 years ago
AUTHORS	Update RocksDB Authors File	7 years ago
CMakeLists.txt	Introduce a mechanism to dump out blocks from block cache and re-insert to secondary cache (#8912 )	3 years ago
CODE_OF_CONDUCT.md	Adopt Contributor Covenant	5 years ago
CONTRIBUTING.md	Add Code of Conduct	7 years ago
COPYING	Add GPLv2 as an alternative license.	8 years ago
DEFAULT_OPTIONS_HISTORY.md	options.delayed_write_rate use the rate of rate_limiter by default.	8 years ago
DUMP_FORMAT.md	First version of rocksdb_dump and rocksdb_undump.	10 years ago
HISTORY.md	Introduce a mechanism to dump out blocks from block cache and re-insert to secondary cache (#8912 )	3 years ago
INSTALL.md	Update installation instructions (#8158 )	4 years ago
LANGUAGE-BINDINGS.md	Update branch name to "main" in README/LANGUAGE_BINDINGS (#8727 )	3 years ago
LICENSE.Apache	Change RocksDB License	8 years ago
LICENSE.leveldb	Add back the LevelDB license file	8 years ago
Makefile	Point bzip2 download address to http://sourceware.org/pub/bzip2 (#8986 )	3 years ago
PLUGINS.md	Add ZenFS to plugin list (#8218 )	4 years ago
README.md	Update branch name to "main" in README/LANGUAGE_BINDINGS (#8727 )	3 years ago
ROCKSDB_LITE.md	Fix some typos in comments and docs.	7 years ago
TARGETS	Introduce a mechanism to dump out blocks from block cache and re-insert to secondary cache (#8912 )	3 years ago
USERS.md	Update USERS.md (#8923 )	3 years ago
Vagrantfile	Adding CentOS 7 Vagrantfile & build script	7 years ago
WINDOWS_PORT.md	Update branch name in WINDOWS_PORT.md (#8745 )	3 years ago
appveyor.yml	Remove 2019 from appveyor (#7038 )	5 years ago
defs.bzl	Make testpilot recognize that these tests have coverage instrumentation	5 years ago
issue_template.md	Add Google Group to Issue Template	5 years ago
src.mk	Introduce a mechanism to dump out blocks from block cache and re-insert to secondary cache (#8912 )	3 years ago
thirdparty.inc	Fix build jemalloc api (#5470 )	6 years ago

README.md

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/main/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.

Design discussions are conducted in https://www.facebook.com/groups/rocksdb.dev/ and https://rocksdb.slack.com/

License

RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.