rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

Peter Dillinger 311a544c2a Use deleters to label cache entries and collect stats (#8297 ) Summary: This change gathers and publishes statistics about the kinds of items in block cache. This is especially important for profiling relative usage of cache by index vs. filter vs. data blocks. It works by iterating over the cache during periodic stats dump (InternalStats, stats_dump_period_sec) or on demand when DB::Get(Map)Property(kBlockCacheEntryStats), except that for efficiency and sharing among column families, saved data from the last scan is used when the data is not considered too old. The new information can be seen in info LOG, for example: Block cache LRUCache@0x7fca62229330 capacity: 95.37 MB collections: 8 last_copies: 0 last_secs: 0.00178 secs_since: 0 Block cache entry stats(count,size,portion): DataBlock(7092,28.24 MB,29.6136%) FilterBlock(215,867.90 KB,0.888728%) FilterMetaBlock(2,5.31 KB,0.00544%) IndexBlock(217,180.11 KB,0.184432%) WriteBuffer(1,256.00 KB,0.262144%) Misc(1,0.00 KB,0%) And also through DB::GetProperty and GetMapProperty (here using ldb just for demonstration): $ ./ldb --db=/dev/shm/dbbench/ get_property rocksdb.block-cache-entry-stats rocksdb.block-cache-entry-stats.bytes.data-block: 0 rocksdb.block-cache-entry-stats.bytes.deprecated-filter-block: 0 rocksdb.block-cache-entry-stats.bytes.filter-block: 0 rocksdb.block-cache-entry-stats.bytes.filter-meta-block: 0 rocksdb.block-cache-entry-stats.bytes.index-block: 178992 rocksdb.block-cache-entry-stats.bytes.misc: 0 rocksdb.block-cache-entry-stats.bytes.other-block: 0 rocksdb.block-cache-entry-stats.bytes.write-buffer: 0 rocksdb.block-cache-entry-stats.capacity: 8388608 rocksdb.block-cache-entry-stats.count.data-block: 0 rocksdb.block-cache-entry-stats.count.deprecated-filter-block: 0 rocksdb.block-cache-entry-stats.count.filter-block: 0 rocksdb.block-cache-entry-stats.count.filter-meta-block: 0 rocksdb.block-cache-entry-stats.count.index-block: 215 rocksdb.block-cache-entry-stats.count.misc: 1 rocksdb.block-cache-entry-stats.count.other-block: 0 rocksdb.block-cache-entry-stats.count.write-buffer: 0 rocksdb.block-cache-entry-stats.id: LRUCache@0x7f3636661290 rocksdb.block-cache-entry-stats.percent.data-block: 0.000000 rocksdb.block-cache-entry-stats.percent.deprecated-filter-block: 0.000000 rocksdb.block-cache-entry-stats.percent.filter-block: 0.000000 rocksdb.block-cache-entry-stats.percent.filter-meta-block: 0.000000 rocksdb.block-cache-entry-stats.percent.index-block: 2.133751 rocksdb.block-cache-entry-stats.percent.misc: 0.000000 rocksdb.block-cache-entry-stats.percent.other-block: 0.000000 rocksdb.block-cache-entry-stats.percent.write-buffer: 0.000000 rocksdb.block-cache-entry-stats.secs_for_last_collection: 0.000052 rocksdb.block-cache-entry-stats.secs_since_last_collection: 0 Solution detail - We need some way to flag what kind of blocks each entry belongs to, preferably without changing the Cache API. One of the complications is that Cache is a general interface that could have other users that don't adhere to whichever convention we decide on for keys and values. Or we would pay for an extra field in the Handle that would only be used for this purpose. This change uses a back-door approach, the deleter, to indicate the "role" of a Cache entry (in addition to the value type, implicitly). This has the added benefit of ensuring proper code origin whenever we recognize a particular role for a cache entry; if the entry came from some other part of the code, it will use an unrecognized deleter, which we simply attribute to the "Misc" role. An internal API makes for simple instantiation and automatic registration of Cache deleters for a given value type and "role". Another internal API, CacheEntryStatsCollector, solves the problem of caching the results of a scan and sharing them, to ensure scans are neither excessive nor redundant so as not to harm Cache performance. Because code is added to BlocklikeTraits, it is pulled out of block_based_table_reader.cc into its own file. This is a reformulation of https://github.com/facebook/rocksdb/issues/8276, without the type checking option (could still be added), and with actual stat gathering. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8297 Test Plan: manual testing with db_bench, and a couple of basic unit tests Reviewed By: ltamasi Differential Revision: D28488721 Pulled By: pdillinger fbshipit-source-id: 472f524a9691b5afb107934be2d41d84f2b129fb		3 years ago
.circleci	Fix cmake failed to build db_bench (#8289 )	3 years ago
.github/workflows	Update clang-format-diff.py path (#7944 )	3 years ago
buckifier	Allow cache_bench/db_bench to use a custom secondary cache (#8312 )	3 years ago
build_tools	Fix crash test with backup as read-only DB (#8161 )	3 years ago
cache	Use deleters to label cache entries and collect stats (#8297 )	3 years ago
cmake	Add `find_dependency()` in cmake config file. (#6791 )	4 years ago
coverage	Find the correct gcov (#6904 )	4 years ago
db	Use deleters to label cache entries and collect stats (#8297 )	3 years ago
db_stress_tool	db_stress: wait for compaction to finish after open with failure injection (#8270 )	3 years ago
docs	Bump rexml from 3.2.4 to 3.2.5 in /docs (#8251 )	3 years ago
env	Fix test issue in new env_test tests (#8319 )	3 years ago
examples	make:Fix c header prototypes (#7994 )	3 years ago
file	Refactor kill point (#8241 )	3 years ago
fuzz	Remove Legacy and Custom FileWrapper classes from header files (#7851 )	3 years ago
hdfs	fix build with 'USE_HDFS' on windows (#6950 )	4 years ago
include/rocksdb	Use deleters to label cache entries and collect stats (#8297 )	3 years ago
java	Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262 )	3 years ago
logging	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033 )	3 years ago
memory	Use thread-safe `strerror_r()` to get error message (#8087 )	3 years ago
memtable	Use deleters to label cache entries and collect stats (#8297 )	3 years ago
monitoring	Disable IOStatsContext/PerfContext if no thread local (#8117 )	3 years ago
options	Refactor Option obj address from char* to void* (#8295 )	3 years ago
plugin	Makefile support to statically link external plugin code (#7918 )	3 years ago
port	Use deleters to label cache entries and collect stats (#8297 )	3 years ago
table	Use deleters to label cache entries and collect stats (#8297 )	3 years ago
test_util	Refactor kill point (#8241 )	3 years ago
third-party	Fix a compilation error in CircleCI vs2019 CXX20 (#8090 )	3 years ago
tools	Use deleters to label cache entries and collect stats (#8297 )	3 years ago
trace_replay	Add request_id in IODebugContext. (#8045 )	3 years ago
util	Add StartThread type checking wrapper (#8303 )	3 years ago
utilities	Use deleters to label cache entries and collect stats (#8297 )	3 years ago
.clang-format	A script that automatically reformat affected lines	11 years ago
.gitignore	gitignore cmake-build-* for CLion integration (#7933 )	3 years ago
.lgtm.yml	Create lgtm.yml for LGTM.com C/C++ analysis (#4058 )	6 years ago
.travis.yml	Move arm build from travis to circleci (#8203 )	3 years ago
.watchmanconfig	Added .watchmanconfig file to rocksdb repo (#5593 )	5 years ago
AUTHORS	Update RocksDB Authors File	7 years ago
CMakeLists.txt	Use deleters to label cache entries and collect stats (#8297 )	3 years ago
CODE_OF_CONDUCT.md	Adopt Contributor Covenant	5 years ago
CONTRIBUTING.md	Add Code of Conduct	7 years ago
COPYING	Add GPLv2 as an alternative license.	7 years ago
DEFAULT_OPTIONS_HISTORY.md	options.delayed_write_rate use the rate of rate_limiter by default.	7 years ago
DUMP_FORMAT.md	First version of rocksdb_dump and rocksdb_undump.	9 years ago
HISTORY.md	Use deleters to label cache entries and collect stats (#8297 )	3 years ago
INSTALL.md	Update installation instructions (#8158 )	3 years ago
LANGUAGE-BINDINGS.md	Add RestoreDBFromLatestBackup to C API, add new C# package (#7092 )	4 years ago
LICENSE.Apache	Change RocksDB License	7 years ago
LICENSE.leveldb	Add back the LevelDB license file	7 years ago
Makefile	Allow cache_bench/db_bench to use a custom secondary cache (#8312 )	3 years ago
PLUGINS.md	Add ZenFS to plugin list (#8218 )	3 years ago
README.md	Fix the CI badge for ppc64le Jenkins (#7561 )	4 years ago
ROCKSDB_LITE.md	Fix some typos in comments and docs.	6 years ago
TARGETS	Use deleters to label cache entries and collect stats (#8297 )	3 years ago
USERS.md	Add Apache Doris to USERS (#7865 )	3 years ago
Vagrantfile	Adding CentOS 7 Vagrantfile & build script	6 years ago
WINDOWS_PORT.md	#5145 , rename port/dirent.h to port/port_dirent.h to avoid compile err when use port dir as header dir output (#5152 )	5 years ago
appveyor.yml	Remove 2019 from appveyor (#7038 )	4 years ago
defs.bzl	Make testpilot recognize that these tests have coverage instrumentation	4 years ago
issue_template.md	Add Google Group to Issue Template	4 years ago
src.mk	Use deleters to label cache entries and collect stats (#8297 )	3 years ago
thirdparty.inc	Fix build jemalloc api (#5470 )	5 years ago

README.md

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/master/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.

Design discussions are conducted in https://www.facebook.com/groups/rocksdb.dev/ and https://rocksdb.slack.com/

License

RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.