rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

Akanksha Mahajan 33f8a08af2 Fix some errors in async prefetching in FilePrefetchBuffer (#9734 ) Summary: In ReadOption `async_io` which prefetches the data asynchronously, db_bench and db_stress runs were failing because wrong data was prefetched which resulted in Error: Checksum mismatched. Wrong data was copied because capacity was less than actual size needed. It has been fixed in this PR. Since there are two separate methods for async and sync prefetching, these changes are in async prefetching methods and any changes would not effect normal prefetching. I ran the regressions to make sure normal prefetching is fine. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9734 Test Plan: 1. CircleCI jobs 2. Ran db_bench ``` . /db_bench -use_existing_db=true -db=/tmp/prefix_scan_prefetch_main -benchmarks="seekrandom" -key_size=32 -value_size=512 -num=5000000 -use_direct_reads=true -seek_nexts=327680 -duration=120 -ops_between_duration_checks=1 -async_io=1 -adaptive_readahead=1 ``` 3. Ran db_stress test ``` export CRASH_TEST_EXT_ARGS=" --async_io=1 --adaptive_readahead=1" make crash_test -j ``` 4. Run regressions for async_io disabled. Old flow without any async changes: ``` ./db_bench -use_existing_db=true -db=/tmp/prefix_scan_prefetch_main -benchmarks="seekrandom" -key_size=32 -value_size=512 -num=5000000 -use_direct_reads=true -seek_nexts=327680 -duration=120 -ops_between_duration_checks=1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags RocksDB: version 7.0 Date: Thu Mar 17 13:11:34 2022 CPU: 24 * Intel Core Processor (Broadwell) CPUCache: 16384 KB Keys: 32 bytes each (+ 0 bytes user-defined timestamp) Values: 512 bytes each (256 bytes after compression) Entries: 5000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 2594.0 MB (estimated) FileSize: 1373.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: Snappy Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/tmp/prefix_scan_prefetch_main] seekrandom : 483618.390 micros/op 2 ops/sec; 338.9 MB/s (249 of 249 found) ``` With async prefetching changes and async_io disabled to make sure in normal prefetching there is no regression. ``` ./db_bench -use_existing_db=true -db=/tmp/prefix_scan_prefetch_main -benchmarks="seekrandom" -key_size=32 -value_size=512 -num=5000000 -use_direct_reads=true -seek_nexts=327680 -duration=120 -ops_between_duration_checks=1 --async_io=0 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags RocksDB: version 7.1 Date: Wed Mar 23 15:56:37 2022 CPU: 24 * Intel Core Processor (Broadwell) CPUCache: 16384 KB Keys: 32 bytes each (+ 0 bytes user-defined timestamp) Values: 512 bytes each (256 bytes after compression) Entries: 5000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 2594.0 MB (estimated) FileSize: 1373.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: Snappy Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/tmp/prefix_scan_prefetch_main] seekrandom : 481819.816 micros/op 2 ops/sec; 340.2 MB/s (250 of 250 found) ``` Reviewed By: riversand963 Differential Revision: D35058471 Pulled By: akankshamahajan15 fbshipit-source-id: 9233a1e6d97cea0c7a8111bfb9e8ac3251c341ce		3 years ago
.circleci	Fix make clean fail after java build (#9710 )	3 years ago
.github/workflows	Use released clang-format instead of the one from dev branch (#9646 )	3 years ago
buckifier	Remove DBGet P95/P99 benchmark metrics (#9742 )	3 years ago
build_tools	Add two new targets to determinator (#9753 )	3 years ago
cache	Update Cache::Release param from force_erase to erase_if_last_ref (#9728 )	3 years ago
cmake	gcc-11 and cmake related cleanup (#9286 )	3 years ago
coverage	Remove asan_symbolize.py for internal asan build (#8737 )	3 years ago
db	Fix heap use-after-free race with DropColumnFamily (#9730 )	3 years ago
db_stress_tool	New backup meta schema, with file temperatures (#9660 )	3 years ago
docs	Update Githubpages version (#9670 )	3 years ago
env	Fix a bug in PosixClock (#9695 )	3 years ago
examples	Remove BlockBasedTableOptions.hash_index_allow_collision (#9454 )	3 years ago
file	Fix some errors in async prefetching in FilePrefetchBuffer (#9734 )	3 years ago
fuzz	Fix compilation errors and add fuzzers to CircleCI (#9420 )	3 years ago
include/rocksdb	Clarify Options::rate_limiter api doc for #9607 Rate-limit automatic WAL flush after each user write (#9745 )	3 years ago
java	jni: uniformly use GetByteArrayRegion() to copy bytes (#9380 )	3 years ago
logging	Use system-wide thread ID in info log lines (#9164 )	3 years ago
memory	Return different Status based on ObjectRegistry::NewObject calls (#9333 )	3 years ago
memtable	Remove using namespace (#9369 )	3 years ago
microbench	Remove DBGet P95/P99 benchmark metrics (#9742 )	3 years ago
monitoring	Dedicate cacheline for DB mutex (#9637 )	3 years ago
options	Fix a major performance bug in 7.0 re: filter compatibility (#9736 )	3 years ago
plugin	Add initial CMake support to plugin (#9214 )	3 years ago
port	#include <winioctl.h> as MSDN prescribes (#9612 )	3 years ago
table	Enable READ_BLOCK_COMPACTION_MICROS to track stats (#9722 )	3 years ago
test_util	Remove BlockBasedTableOptions.hash_index_allow_collision (#9454 )	3 years ago
third-party	Work around some new clang-analyze failures (#9515 )	3 years ago
tools	Correctly set ThreadState::tid (#9757 )	3 years ago
trace_replay	Added `TraceOptions::preserve_write_order` (#9334 )	3 years ago
util	Add manifest fix-up utility for file temperatures (#9683 )	3 years ago
utilities	Update Cache::Release param from force_erase to erase_if_last_ref (#9728 )	3 years ago
.clang-format	A script that automatically reformat affected lines	11 years ago
.gitignore	gitignore cmake-build-* for CLion integration (#7933 )	4 years ago
.lgtm.yml	Create lgtm.yml for LGTM.com C/C++ analysis (#4058 )	7 years ago
.travis.yml	Reenable s390x platform_dependent travis job (#9631 )	3 years ago
.watchmanconfig	Added .watchmanconfig file to rocksdb repo (#5593 )	6 years ago
AUTHORS	Update RocksDB Authors File	7 years ago
CMakeLists.txt	Avoid popcnt on Windows when unavailable and in portable builds (#9680 )	3 years ago
CODE_OF_CONDUCT.md	Adopt Contributor Covenant	5 years ago
CONTRIBUTING.md	Add Code of Conduct	7 years ago
COPYING	Add GPLv2 as an alternative license.	8 years ago
DEFAULT_OPTIONS_HISTORY.md	Add Options::DisableExtraChecks, clarify force_consistency_checks (#9363 )	3 years ago
DUMP_FORMAT.md	First version of rocksdb_dump and rocksdb_undump.	10 years ago
HISTORY.md	Enable READ_BLOCK_COMPACTION_MICROS to track stats (#9722 )	3 years ago
INSTALL.md	Clarify Google benchmark < 1.6.0 in INSTALL.md (#9505 )	3 years ago
LANGUAGE-BINDINGS.md	Update branch name to "main" in README/LANGUAGE_BINDINGS (#8727 )	3 years ago
LICENSE.Apache	Change RocksDB License	8 years ago
LICENSE.leveldb	Add back the LevelDB license file	8 years ago
Makefile	Fix make clean fail after java build (#9710 )	3 years ago
PLUGINS.md	Move RADOS support to separate repo (#9206 )	3 years ago
README.md	README: De-list slack channel, list Google group (#9387 )	3 years ago
ROCKSDB_LITE.md	Fix some typos in comments and docs.	7 years ago
TARGETS	Remove DBGet P95/P99 benchmark metrics (#9742 )	3 years ago
USERS.md	Add Solana's RocksDB use case in USERS.md (#9558 )	3 years ago
Vagrantfile	Adding CentOS 7 Vagrantfile & build script	7 years ago
WINDOWS_PORT.md	Update branch name in WINDOWS_PORT.md (#8745 )	3 years ago
crash_test.mk	Add new determinators for multiops transactions stress test (#9708 )	3 years ago
issue_template.md	Add Google Group to Issue Template	5 years ago
python.mk	crash_test Makefile refactoring, add to CircleCI (#9702 )	3 years ago
src.mk	Support user-defined timestamps in write-committed txns (#9629 )	3 years ago
thirdparty.inc	Fix build jemalloc api (#5470 )	6 years ago

README.md

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/main/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.

Questions and discussions are welcome on the RocksDB Developers Public Facebook group and email list on Google Groups.

License

RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.