rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

Hui Xiao 74544d582f Account Bloom/Ribbon filter construction memory in global memory limit (#9073 ) Summary: Note: This PR is the 4th part of a bigger PR stack (https://github.com/facebook/rocksdb/pull/9073) and will rebase/merge only after the first three PRs (https://github.com/facebook/rocksdb/pull/9070, https://github.com/facebook/rocksdb/pull/9071, https://github.com/facebook/rocksdb/pull/9130) merge. Context: Similar to https://github.com/facebook/rocksdb/pull/8428, this PR is to track memory usage during (new) Bloom Filter (i.e,FastLocalBloom) and Ribbon Filter (i.e, Ribbon128) construction, moving toward the goal of [single global memory limit using block cache capacity](https://github.com/facebook/rocksdb/wiki/Projects-Being-Developed#improving-memory-efficiency). It also constrains the size of the banding portion of Ribbon Filter during construction by falling back to Bloom Filter if that banding is, at some point, larger than the available space in the cache under `LRUCacheOptions::strict_capacity_limit=true`. The option to turn on this feature is `BlockBasedTableOptions::reserve_table_builder_memory = true` which by default is set to `false`. We [decided](https://github.com/facebook/rocksdb/pull/9073#discussion_r741548409) not to have separate option for separate memory user in table building therefore their memory accounting are all bundled under one general option. Summary: - Reserved/released cache for creation/destruction of three main memory users with the passed-in `FilterBuildingContext::cache_res_mgr` during filter construction: - hash entries (i.e`hash_entries`.size(), we bucket-charge hash entries during insertion for performance), - banding (Ribbon Filter only, `bytes_coeff_rows` +`bytes_result_rows` + `bytes_backtrack`), - final filter (i.e, `mutable_buf`'s size). - Implementation details: in order to use `CacheReservationManager::CacheReservationHandle` to account final filter's memory, we have to store the `CacheReservationManager` object and `CacheReservationHandle` for final filter in `XXPH3BitsFilterBuilder` as well as explicitly delete the filter bits builder when done with the final filter in block based table. - Added option fo run `filter_bench` with this memory reservation feature Pull Request resolved: https://github.com/facebook/rocksdb/pull/9073 Test Plan: - Added new tests in `db_bloom_filter_test` to verify filter construction peak cache reservation under combination of `BlockBasedTable::Rep::FilterType` (e.g, `kFullFilter`, `kPartitionedFilter`), `BloomFilterPolicy::Mode`(e.g, `kFastLocalBloom`, `kStandard128Ribbon`, `kDeprecatedBlock`) and `BlockBasedTableOptions::reserve_table_builder_memory` - To address the concern for slow test: tests with memory reservation under `kFullFilter` + `kStandard128Ribbon` and `kPartitionedFilter` take around 3000 - 6000 ms and others take around 1500 - 2000 ms, in total adding 20000 - 25000 ms to the test suit running locally - Added new test in `bloom_test` to verify Ribbon Filter fallback on large banding in FullFilter - Added test in `filter_bench` to verify that this feature does not significantly slow down Bloom/Ribbon Filter construction speed. Local result averaged over 20 run as below: - FastLocalBloom - baseline `./filter_bench -impl=2 -quick -runs 20 \| grep 'Build avg'`: - Build avg ns/key: 29.56295 (DEBUG_LEVEL=1), 29.98153 (DEBUG_LEVEL=0) - new feature (expected to be similar as above)`./filter_bench -impl=2 -quick -runs 20 -reserve_table_builder_memory=true \| grep 'Build avg'`: - Build avg ns/key: 30.99046 (DEBUG_LEVEL=1), 30.48867 (DEBUG_LEVEL=0) - new feature of RibbonFilter with fallback (expected to be similar as above) `./filter_bench -impl=2 -quick -runs 20 -reserve_table_builder_memory=true -strict_capacity_limit=true \| grep 'Build avg'` : - Build avg ns/key: 31.146975 (DEBUG_LEVEL=1), 30.08165 (DEBUG_LEVEL=0) - Ribbon128 - baseline `./filter_bench -impl=3 -quick -runs 20 \| grep 'Build avg'`: - Build avg ns/key: 129.17585 (DEBUG_LEVEL=1), 130.5225 (DEBUG_LEVEL=0) - new feature (expected to be similar as above) `./filter_bench -impl=3 -quick -runs 20 -reserve_table_builder_memory=true \| grep 'Build avg' `: - Build avg ns/key: 131.61645 (DEBUG_LEVEL=1), 132.98075 (DEBUG_LEVEL=0) - new feature of RibbonFilter with fallback (expected to be a lot faster than above due to fallback) `./filter_bench -impl=3 -quick -runs 20 -reserve_table_builder_memory=true -strict_capacity_limit=true \| grep 'Build avg'` : - Build avg ns/key: 52.032965 (DEBUG_LEVEL=1), 52.597825 (DEBUG_LEVEL=0) - And the warning message of `"Cache reservation for Ribbon filter banding failed due to cache full"` is indeed logged to console. Reviewed By: pdillinger Differential Revision: D31991348 Pulled By: hx235 fbshipit-source-id: 9336b2c60f44d530063da518ceaf56dac5f9df8e		3 years ago
.circleci	Fix EnvLibrados and add to CI (#9088 )	3 years ago
.github/workflows	Add (& fix) some simple source code checks (#8821 )	3 years ago
buckifier	Update buckify scripts (#9104 )	3 years ago
build_tools	Parallelize sandcastle tests (#9178 )	3 years ago
cache	Account Bloom/Ribbon filter construction memory in global memory limit (#9073 )	3 years ago
cmake	Add `find_dependency()` in cmake config file. (#6791 )	4 years ago
coverage	Remove asan_symbolize.py for internal asan build (#8737 )	3 years ago
db	Account Bloom/Ribbon filter construction memory in global memory limit (#9073 )	3 years ago
db_stress_tool	Secondary cache error injection (#9002 )	3 years ago
docs	Misc doc fixes (#8983 )	3 years ago
env	Use system-wide thread ID in info log lines (#9164 )	3 years ago
examples	Add (& fix) some simple source code checks (#8821 )	3 years ago
file	Add tiered storage related read bytes stats to Statistic (#9123 )	3 years ago
fuzz	Make EventListener into a Customizable Class (#8473 )	3 years ago
hdfs	fix build with 'USE_HDFS' on windows (#6950 )	4 years ago
include/rocksdb	Account Bloom/Ribbon filter construction memory in global memory limit (#9073 )	3 years ago
java	Fix integer overflow in TraceOptions (#9157 )	3 years ago
logging	Use system-wide thread ID in info log lines (#9164 )	3 years ago
memory	Fix and detect headers with missing dependencies (#8893 )	3 years ago
memtable	Minor improvement to CacheReservationManager/WriteBufferManager/CompressionDictBuilding (#9139 )	3 years ago
microbench	Skip directory fsync for filesystem btrfs (#8903 )	3 years ago
monitoring	Add tiered storage related read bytes stats to Statistic (#9123 )	3 years ago
options	Account Bloom/Ribbon filter construction memory in global memory limit (#9073 )	3 years ago
plugin	Makefile support to statically link external plugin code (#7918 )	3 years ago
port	Fix typo in env_win.h (#9138 )	3 years ago
table	Account Bloom/Ribbon filter construction memory in global memory limit (#9073 )	3 years ago
test_util	Enable a few unit tests to use custom Env objects (#9087 )	3 years ago
third-party	Add support for building on s390x platform (#8962 )	3 years ago
tools	Reuse internal auto readhead_size at each Level (expect L0) for Iterations (#9056 )	3 years ago
trace_replay	Cleanup includes in dbformat.h (#8930 )	3 years ago
util	Account Bloom/Ribbon filter construction memory in global memory limit (#9073 )	3 years ago
utilities	Fix BackupEngine's internal callers of GenericRateLimiter::Request() not honoring bytes <= GetSingleBurstBytes() (#9063 )	3 years ago
.clang-format	A script that automatically reformat affected lines	11 years ago
.gitignore	gitignore cmake-build-* for CLion integration (#7933 )	3 years ago
.lgtm.yml	Create lgtm.yml for LGTM.com C/C++ analysis (#4058 )	6 years ago
.travis.yml	Re-enable 390x+cmake* Travis jobs (#9110 )	3 years ago
.watchmanconfig	Added .watchmanconfig file to rocksdb repo (#5593 )	5 years ago
AUTHORS	Update RocksDB Authors File	7 years ago
CMakeLists.txt	Secondary cache error injection (#9002 )	3 years ago
CODE_OF_CONDUCT.md	Adopt Contributor Covenant	5 years ago
CONTRIBUTING.md	Add Code of Conduct	7 years ago
COPYING	Add GPLv2 as an alternative license.	7 years ago
DEFAULT_OPTIONS_HISTORY.md	options.delayed_write_rate use the rate of rate_limiter by default.	7 years ago
DUMP_FORMAT.md	First version of rocksdb_dump and rocksdb_undump.	9 years ago
HISTORY.md	Account Bloom/Ribbon filter construction memory in global memory limit (#9073 )	3 years ago
INSTALL.md	Update installation instructions (#8158 )	3 years ago
LANGUAGE-BINDINGS.md	Update branch name to "main" in README/LANGUAGE_BINDINGS (#8727 )	3 years ago
LICENSE.Apache	Change RocksDB License	7 years ago
LICENSE.leveldb	Add back the LevelDB license file	7 years ago
Makefile	Don't allow parallel crash_test in Makefile (#9180 )	3 years ago
PLUGINS.md	Add ZenFS to plugin list (#8218 )	3 years ago
README.md	Update branch name to "main" in README/LANGUAGE_BINDINGS (#8727 )	3 years ago
ROCKSDB_LITE.md	Fix some typos in comments and docs.	6 years ago
TARGETS	Secondary cache error injection (#9002 )	3 years ago
USERS.md	Update USERS.md (#8923 )	3 years ago
Vagrantfile	Adding CentOS 7 Vagrantfile & build script	6 years ago
WINDOWS_PORT.md	Update branch name in WINDOWS_PORT.md (#8745 )	3 years ago
appveyor.yml	Remove 2019 from appveyor (#7038 )	4 years ago
defs.bzl	Make testpilot recognize that these tests have coverage instrumentation	4 years ago
issue_template.md	Add Google Group to Issue Template	4 years ago
src.mk	Secondary cache error injection (#9002 )	3 years ago
thirdparty.inc	Fix build jemalloc api (#5470 )	5 years ago

README.md

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/main/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.

Design discussions are conducted in https://www.facebook.com/groups/rocksdb.dev/ and https://rocksdb.slack.com/

License

RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.