rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

Changyu Bi 8515bd50c9 Support read rate-limiting in SequentialFileReader (#9973 ) Summary: Added rate limiter and read rate-limiting support to SequentialFileReader. I've updated call sites to SequentialFileReader::Read with appropriate IO priority (or left a TODO and specified IO_TOTAL for now). The PR is separated into four commits: the first one added the rate-limiting support, but with some fixes in the unit test since the number of request bytes from rate limiter in SequentialFileReader are not accurate (there is overcharge at EOF). The second commit fixed this by allowing SequentialFileReader to check file size and determine how many bytes are left in the file to read. The third commit added benchmark related code. The fourth commit moved the logic of using file size to avoid overcharging the rate limiter into backup engine (the main user of SequentialFileReader). Pull Request resolved: https://github.com/facebook/rocksdb/pull/9973 Test Plan: - `make check`, backup_engine_test covers usage of SequentialFileReader with rate limiter. - Run db_bench to check if rate limiting is throttling as expected: Verified that reads and writes are together throttled at 2MB/s, and at 0.2MB chunks that are 100ms apart. - Set up: `./db_bench --benchmarks=fillrandom -db=/dev/shm/test_rocksdb` - Benchmark: ``` strace -ttfe read,write ./db_bench --benchmarks=backup -db=/dev/shm/test_rocksdb --backup_rate_limit=2097152 --use_existing_db strace -ttfe read,write ./db_bench --benchmarks=restore -db=/dev/shm/test_rocksdb --restore_rate_limit=2097152 --use_existing_db ``` - db bench on backup and restore to ensure no performance regression. - backup (avg over 50 runs): pre-change: 1.90443e+06 micros/op; post-change: 1.8993e+06 micros/op (improve by 0.2%) - restore (avg over 50 runs): pre-change: 1.79105e+06 micros/op; post-change: 1.78192e+06 micros/op (improve by 0.5%) ``` # Set up ./db_bench --benchmarks=fillrandom -db=/tmp/test_rocksdb -num=10000000 # benchmark TEST_TMPDIR=/tmp/test_rocksdb NUM_RUN=50 for ((j=0;j<$NUM_RUN;j++)) do ./db_bench -db=$TEST_TMPDIR -num=10000000 -benchmarks=backup -use_existing_db \| egrep 'backup' # Restore #./db_bench -db=$TEST_TMPDIR -num=10000000 -benchmarks=restore -use_existing_db done > rate_limit.txt && awk -v NUM_RUN=$NUM_RUN '{sum+=$3;sum_sqrt+=$3^2}END{print sum/NUM_RUN, sqrt(sum_sqrt/NUM_RUN-(sum/NUM_RUN)^2)}' rate_limit.txt >> rate_limit_2.txt ``` Reviewed By: hx235 Differential Revision: D36327418 Pulled By: cbi42 fbshipit-source-id: e75d4307cff815945482df5ba630c1e88d064691		2 years ago
.circleci	Point libprotobuf-mutator to the latest verified commit hash (#10028 )	2 years ago
.github/workflows	Use released clang-format instead of the one from dev branch (#9646 )	3 years ago
buckifier	Multi file concurrency in MultiGet using coroutines and async IO (#9968 )	3 years ago
build_tools	Minimum macOS version needed to build v7.2.2 and up is 10.13 (#9976 )	2 years ago
cache	Remove own ToString() (#9955 )	3 years ago
cmake	gcc-11 and cmake related cleanup (#9286 )	3 years ago
coverage	Fix commit_prereq and other targets (#9797 )	3 years ago
db	Support read rate-limiting in SequentialFileReader (#9973 )	2 years ago
db_stress_tool	Support using ZDICT_finalizeDictionary to generate zstd dictionary (#9857 )	2 years ago
docs	Bump nokogiri from 1.13.4 to 1.13.6 in /docs (#10019 )	2 years ago
env	Support read rate-limiting in SequentialFileReader (#9973 )	2 years ago
examples	Remove BlockBasedTableOptions.hash_index_allow_collision (#9454 )	3 years ago
file	Support read rate-limiting in SequentialFileReader (#9973 )	2 years ago
fuzz	Fix compilation errors and add fuzzers to CircleCI (#9420 )	3 years ago
include/rocksdb	Expose unix time in rocksdb::Snapshot (#9923 )	2 years ago
java	Java build: finish compiling before testing (etc) (#10034 )	2 years ago
logging	Use system-wide thread ID in info log lines (#9164 )	3 years ago
memory	Remove ROCKSDB_SUPPORT_THREAD_LOCAL define because it's a part of C++11 (#10015 )	3 years ago
memtable	Rewrite memory-charging feature's option API (#9926 )	3 years ago
microbench	Add microbenchmarks for `DB::GetMergeOperands()` (#9971 )	3 years ago
monitoring	Seek parallelization (#9994 )	2 years ago
options	Support read rate-limiting in SequentialFileReader (#9973 )	2 years ago
plugin	Add initial CMake support to plugin (#9214 )	3 years ago
port	Remove ROCKSDB_SUPPORT_THREAD_LOCAL define because it's a part of C++11 (#10015 )	3 years ago
table	Fix stress test failure "Corruption: checksum mismatch" or "Iterator Diverged" with async_io enabled (#10032 )	2 years ago
test_util	Fix a bug of not setting enforce_single_del_contracts (#10027 )	2 years ago
third-party	Meta-internal folly integration with F14FastMap (#9546 )	3 years ago
tools	Support read rate-limiting in SequentialFileReader (#9973 )	2 years ago
trace_replay	Use std::numeric_limits<> (#9954 )	3 years ago
util	Support read rate-limiting in SequentialFileReader (#9973 )	2 years ago
utilities	Support read rate-limiting in SequentialFileReader (#9973 )	2 years ago
.clang-format	A script that automatically reformat affected lines	11 years ago
.gitignore	Add timestamp support to DBImplReadOnly (#10004 )	3 years ago
.lgtm.yml	Create lgtm.yml for LGTM.com C/C++ analysis (#4058 )	6 years ago
.travis.yml	Fix remaining uses of "backupable" (#9792 )	3 years ago
.watchmanconfig	Added .watchmanconfig file to rocksdb repo (#5593 )	5 years ago
AUTHORS	Update RocksDB Authors File	7 years ago
CMakeLists.txt	Add plugin header install in CMakeLists.txt (#10025 )	2 years ago
CODE_OF_CONDUCT.md	Adopt Contributor Covenant	5 years ago
CONTRIBUTING.md	Add Code of Conduct	7 years ago
COPYING	Add GPLv2 as an alternative license.	8 years ago
DEFAULT_OPTIONS_HISTORY.md	Add Options::DisableExtraChecks, clarify force_consistency_checks (#9363 )	3 years ago
DUMP_FORMAT.md	First version of rocksdb_dump and rocksdb_undump.	10 years ago
HISTORY.md	Expose unix time in rocksdb::Snapshot (#9923 )	2 years ago
INSTALL.md	Update supported VS versions in INSTALL.md (#9823 )	3 years ago
LANGUAGE-BINDINGS.md	Update branch name to "main" in README/LANGUAGE_BINDINGS (#8727 )	3 years ago
LICENSE.Apache	Change RocksDB License	7 years ago
LICENSE.leveldb	Add back the LevelDB license file	7 years ago
Makefile	Java build: finish compiling before testing (etc) (#10034 )	2 years ago
PLUGINS.md	Add pmem-rocksdb-plugin link in PLUGINs.md (#9934 )	3 years ago
README.md	README: De-list slack channel, list Google group (#9387 )	3 years ago
ROCKSDB_LITE.md	Fix remaining uses of "backupable" (#9792 )	3 years ago
TARGETS	Add timestamp support to DBImplReadOnly (#10004 )	3 years ago
USERS.md	Add Solana's RocksDB use case in USERS.md (#9558 )	3 years ago
Vagrantfile	Adding CentOS 7 Vagrantfile & build script	7 years ago
WINDOWS_PORT.md	Update branch name in WINDOWS_PORT.md (#8745 )	3 years ago
common.mk	Clean up variables for temporary directory (#9961 )	3 years ago
crash_test.mk	Clean up variables for temporary directory (#9961 )	3 years ago
issue_template.md	Add Google Group to Issue Template	5 years ago
rocksdb.pc.in	Generate pkg-config file via CMake (#9945 )	3 years ago
src.mk	Add timestamp support to DBImplReadOnly (#10004 )	3 years ago
thirdparty.inc	Fix build jemalloc api (#5470 )	5 years ago

README.md

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/main/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.

Questions and discussions are welcome on the RocksDB Developers Public Facebook group and email list on Google Groups.

License

RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.