rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

Bo Wang 87b82f28a1 Split cache to minimize internal fragmentation (#10287 ) Summary: ### Summary: To minimize the internal fragmentation caused by the variable size of the compressed blocks, the original block is split according to the jemalloc bin size in `Insert()` and then merged back in `Lookup()`. Based on the analysis of the results of the following tests, from the overall internal fragmentation perspective, this PR does mitigate the internal fragmentation issue. _Do more myshadow tests with the latest commit. I finished several myshadow AB Testing and the results are promising. For the config of 4GB primary cache and 3GB secondary cache, Jemalloc resident stats shows consistently ~0.15GB memory saving; the allocated and active stats show similar memory savings. The CPU usage is almost the same before and after this PR._ To evaluate the issue of memory fragmentations and the benefits of this PR, I conducted two sets of local tests as follows. T1 Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 100 bytes each (50 bytes after compression) Entries: 90000000 RawSize: 9956.4 MB (estimated) FileSize: 5664.8 MB (estimated) \| Test Name \| Primary Cache Size (MB) \| Compressed Secondary Cache Size (MB) \| \| - \| - \| - \| \| T1_3 \| 4000 \| 4000 \| \| T1_4 \| 2000 \| 3000 \| Populate the DB: ./db_bench --benchmarks=fillrandom --num=90000000 -db=/mem_fragmentation/db_bench_1 Overwrite it to a stable state: ./db_bench --benchmarks=overwrite --num=90000000 -use_existing_db -db=/mem_fragmentation/db_bench_1 Run read tests with differnt cache setting: T1_3: MALLOC_CONF="prof:true,prof_stats:true" ../rocksdb/db_bench --benchmarks=seekrandom --threads=16 --num=90000000 -use_existing_db --benchmark_write_rate_limit=52000000 -use_direct_reads --cache_size=4000000000 -compressed_secondary_cache_size=4000000000 -use_compressed_secondary_cache -db=/mem_fragmentation/db_bench_1 --print_malloc_stats=true > ~/temp/mem_frag/20220710/jemalloc_stats_json_T1_3_20220710 -duration=1800 & T1_4: MALLOC_CONF="prof:true,prof_stats:true" ../rocksdb/db_bench --benchmarks=seekrandom --threads=16 --num=90000000 -use_existing_db --benchmark_write_rate_limit=52000000 -use_direct_reads --cache_size=2000000000 -compressed_secondary_cache_size=3000000000 -use_compressed_secondary_cache -db=/mem_fragmentation/db_bench_1 --print_malloc_stats=true > ~/temp/mem_frag/20220710/jemalloc_stats_json_T1_4_20220710 -duration=1800 & For T1_3 and T1_4, I also conducted the tests before and after this PR. The following table show the important jemalloc stats. \| Test Name \| T1_3 \| T1_3 after mem defrag \| T1_4 \| T1_4 after mem defrag \| \| - \| - \| - \| - \| - \| \| allocated (MB) \| 8728 \| 8076 \| 5518 \| 5043 \| \| available (MB) \| 8753 \| 8092 \| 5536 \| 5051 \| \| external fragmentation rate \| 0.003 \| 0.002 \| 0.003 \| 0.0016 \| \| resident (MB) \| 8956 \| 8365 \| 5655 \| 5235 \| T2 Keys: 32 bytes each (+ 0 bytes user-defined timestamp) Values: 256 bytes each (128 bytes after compression) Entries: 40000000 RawSize: 10986.3 MB (estimated) FileSize: 6103.5 MB (estimated) \| Test Name \| Primary Cache Size (MB) \| Compressed Secondary Cache Size (MB) \| \| - \| - \| - \| \| T2_3 \| 4000 \| 4000 \| \| T2_4 \| 2000 \| 3000 \| Create DB (10GB): ./db_bench -benchmarks=fillrandom -use_direct_reads=true -num=40000000 -key_size=32 -value_size=256 -db=/mem_fragmentation/db_bench_2 Overwrite it to a stable state: ./db_bench --benchmarks=overwrite --num=40000000 -use_existing_db -key_size=32 -value_size=256 -db=/mem_fragmentation/db_bench_2 Run read tests with differnt cache setting: T2_3: MALLOC_CONF="prof:true,prof_stats:true" ./db_bench --benchmarks="mixgraph" -use_direct_io_for_flush_and_compaction=true -use_direct_reads=true -cache_size=4000000000 -compressed_secondary_cache_size=4000000000 -use_compressed_secondary_cache -keyrange_dist_a=14.18 -keyrange_dist_b=-2.917 -keyrange_dist_c=0.0164 -keyrange_dist_d=-0.08082 -keyrange_num=30 -value_k=0.2615 -value_sigma=25.45 -iter_k=2.517 -iter_sigma=14.236 -mix_get_ratio=0.85 -mix_put_ratio=0.14 -mix_seek_ratio=0.01 -sine_mix_rate_interval_milliseconds=5000 -sine_a=1000 -sine_b=0.000073 -sine_d=400000 -reads=80000000 -num=40000000 -key_size=32 -value_size=256 -use_existing_db=true -db=/mem_fragmentation/db_bench_2 --print_malloc_stats=true > ~/temp/mem_frag/jemalloc_stats_T2_3 -duration=1800 & T2_4: MALLOC_CONF="prof:true,prof_stats:true" ./db_bench --benchmarks="mixgraph" -use_direct_io_for_flush_and_compaction=true -use_direct_reads=true -cache_size=2000000000 -compressed_secondary_cache_size=3000000000 -use_compressed_secondary_cache -keyrange_dist_a=14.18 -keyrange_dist_b=-2.917 -keyrange_dist_c=0.0164 -keyrange_dist_d=-0.08082 -keyrange_num=30 -value_k=0.2615 -value_sigma=25.45 -iter_k=2.517 -iter_sigma=14.236 -mix_get_ratio=0.85 -mix_put_ratio=0.14 -mix_seek_ratio=0.01 -sine_mix_rate_interval_milliseconds=5000 -sine_a=1000 -sine_b=0.000073 -sine_d=400000 -reads=80000000 -num=40000000 -key_size=32 -value_size=256 -use_existing_db=true -db=/mem_fragmentation/db_bench_2 --print_malloc_stats=true > ~/temp/mem_frag/jemalloc_stats_T2_4 -duration=1800 & For T2_3 and T2_4, I also conducted the tests before and after this PR. The following table show the important jemalloc stats. \| Test Name \| T2_3 \| T2_3 after mem defrag \| T2_4 \| T2_4 after mem defrag \| \| - \| - \| - \| - \| - \| \| allocated (MB) \| 8425 \| 8093 \| 5426 \| 5149 \| \| available (MB) \| 8489 \| 8138 \| 5435 \| 5158 \| \| external fragmentation rate \| 0.008 \| 0.0055 \| 0.0017 \| 0.0017 \| \| resident (MB) \| 8676 \| 8392 \| 5541 \| 5321 \| Pull Request resolved: https://github.com/facebook/rocksdb/pull/10287 Test Plan: Unit tests. Reviewed By: anand1976 Differential Revision: D37743362 Pulled By: gitbw95 fbshipit-source-id: 0010c5af08addeacc5ebbc4ffe5be882fb1d38ad		2 years ago
.circleci	Run new benchmark script in branch. (#10303 )	2 years ago
.github/workflows	Use released clang-format instead of the one from dev branch (#9646 )	3 years ago
buckifier	Use optimized folly DistributedMutex in LRUCache when available (#10179 )	2 years ago
build_tools	Run new benchmark script in branch. (#10303 )	2 years ago
cache	Split cache to minimize internal fragmentation (#10287 )	2 years ago
cmake	gcc-11 and cmake related cleanup (#9286 )	3 years ago
coverage	Fix commit_prereq and other targets (#9797 )	3 years ago
db	Fix race in ExitAsBatchGroupLeader with pipelined writes (#9944 )	2 years ago
db_stress_tool	Stop operating on DB in a stress test background thread (#10373 )	2 years ago
docs	Bump tzinfo from 1.2.9 to 1.2.10 in /docs (#10400 )	2 years ago
env	Fix serious FSDirectory use-after-Close bug (missing fsync) (#10460 )	2 years ago
examples	Add a simple example of backup and restore (#10054 )	2 years ago
file	Provide support for direct_reads with async_io (#10197 )	2 years ago
fuzz	Fix compilation errors and add fuzzers to CircleCI (#9420 )	3 years ago
include/rocksdb	Use EnvLogger instead of PosixLogger (#10436 )	2 years ago
java	Revert "Add a blob-specific cache priority (#10309 )" (#10434 )	2 years ago
logging	Use EnvLogger instead of PosixLogger (#10436 )	2 years ago
memory	Revert "Add a blob-specific cache priority (#10309 )" (#10434 )	2 years ago
memtable	Fix a bug in hash linked list (#10401 )	2 years ago
microbench	Fix overflow in ribbon_bench after #10184 (#10195 )	2 years ago
monitoring	Use EnvLogger instead of PosixLogger (#10436 )	2 years ago
options	Fix race conditions in GenericRateLimiter (#10374 )	2 years ago
plugin	Add initial CMake support to plugin (#9214 )	3 years ago
port	Use EnvLogger instead of PosixLogger (#10436 )	2 years ago
table	Fix cache metrics update when secondary cache is used (#10440 )	2 years ago
test_util	Add seqno to time mapping (#10338 )	2 years ago
third-party	Meta-internal folly integration with F14FastMap (#9546 )	3 years ago
tools	regression_test.sh: kill very old db_bench (and more) (#10441 )	2 years ago
trace_replay	Use std::numeric_limits<> (#9954 )	3 years ago
util	db_bench -use_stderr_info_logger to print timestamp (#10435 )	2 years ago
utilities	Fix serious FSDirectory use-after-Close bug (missing fsync) (#10460 )	2 years ago
.clang-format	A script that automatically reformat affected lines	11 years ago
.gitignore	Snapshots with user-specified timestamps (#9879 )	2 years ago
.lgtm.yml	Create lgtm.yml for LGTM.com C/C++ analysis (#4058 )	6 years ago
.watchmanconfig	Added .watchmanconfig file to rocksdb repo (#5593 )	5 years ago
AUTHORS	Update RocksDB Authors File	7 years ago
CMakeLists.txt	add trace tools flags in CMakeLists (#10404 )	2 years ago
CODE_OF_CONDUCT.md	Adopt Contributor Covenant	5 years ago
CONTRIBUTING.md	Add Code of Conduct	7 years ago
COPYING	Add GPLv2 as an alternative license.	8 years ago
DEFAULT_OPTIONS_HISTORY.md	Add Options::DisableExtraChecks, clarify force_consistency_checks (#9363 )	3 years ago
DUMP_FORMAT.md	First version of rocksdb_dump and rocksdb_undump.	10 years ago
HISTORY.md	Split cache to minimize internal fragmentation (#10287 )	2 years ago
INSTALL.md	Update supported VS versions in INSTALL.md (#9823 )	3 years ago
LANGUAGE-BINDINGS.md	Update branch name to "main" in README/LANGUAGE_BINDINGS (#8727 )	3 years ago
LICENSE.Apache	Change RocksDB License	7 years ago
LICENSE.leveldb	Add back the LevelDB license file	7 years ago
Makefile	Add seqno to time mapping (#10338 )	2 years ago
PLUGINS.md	Add pmem-rocksdb-plugin link in PLUGINs.md (#9934 )	3 years ago
README.md	Remove Travis CI (#10407 )	2 years ago
ROCKSDB_LITE.md	Fix remaining uses of "backupable" (#9792 )	3 years ago
TARGETS	Charge blob cache usage against the global memory limit (#10321 )	2 years ago
USERS.md	Add Solana's RocksDB use case in USERS.md (#9558 )	3 years ago
Vagrantfile	Adding CentOS 7 Vagrantfile & build script	7 years ago
WINDOWS_PORT.md	Update branch name in WINDOWS_PORT.md (#8745 )	3 years ago
common.mk	Clean up variables for temporary directory (#9961 )	3 years ago
crash_test.mk	Clean up variables for temporary directory (#9961 )	3 years ago
issue_template.md	Add Google Group to Issue Template	5 years ago
rocksdb.pc.in	build: fix pkg-config file generation (#9953 )	2 years ago
src.mk	Charge blob cache usage against the global memory limit (#10321 )	2 years ago
thirdparty.inc	Fix build jemalloc api (#5470 )	5 years ago

README.md

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/main/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.

Questions and discussions are welcome on the RocksDB Developers Public Facebook group and email list on Google Groups.

License

RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.