rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

Sagar Vemuri d938226af4 Improve performance of long range scans with readahead Summary: This change improves the performance of iterators doing long range scans (e.g. big/full table scans in MyRocks) by using readahead and prefetching additional data on each disk IO. This prefetching is automatically enabled on noticing more than 2 IOs for the same table file during iteration. The readahead size starts with 8KB and is exponentially increased on each additional sequential IO, up to a max of 256 KB. This helps in cutting down the number of IOs needed to complete the range scan. Constraints: - The prefetched data is stored by the OS in page cache. So this currently works only for non direct-reads use-cases i.e applications which use page cache. (Direct-I/O support will be enabled in a later PR). - This gets currently enabled only when ReadOptions.readahead_size = 0 (which is the default value). Thanks to siying for the original idea and implementation. Benchmarks: Data fill: ``` TEST_TMPDIR=/data/users/$USER/benchmarks/iter ./db_bench -benchmarks=fillrandom -num=1000000000 -compression_type="none" -level_compaction_dynamic_level_bytes ``` Do a long range scan: Seekrandom with large number of nexts ``` TEST_TMPDIR=/data/users/$USER/benchmarks/iter ./db_bench -benchmarks=seekrandom -duration=60 -num=1000000000 -use_existing_db -seek_nexts=10000 -statistics -histogram ``` Page cache was cleared before each experiment with the command: ``` sudo sh -c "echo 3 > /proc/sys/vm/drop_caches" ``` ``` Before: seekrandom : 34020.945 micros/op 29 ops/sec; 32.5 MB/s (1636 of 1999 found) With this change: seekrandom : 8726.912 micros/op 114 ops/sec; 126.8 MB/s (5702 of 6999 found) ``` ~3.9X performance improvement. Also verified with strace and gdb that the readahead size is increasing as expected. ``` strace -e readahead -f -T -t -p <db_bench process pid> ``` Closes https://github.com/facebook/rocksdb/pull/3282 Differential Revision: D6586477 Pulled By: sagar0 fbshipit-source-id: 8a118a0ed4594fbb7f5b1cafb242d7a4033cb58c		7 years ago
buckifier	Remove `import` use from TARGETS	7 years ago
build_tools	FreeBSD build support for RocksDB and RocksJava	7 years ago
cache	fix gflags namespace	7 years ago
cmake	add missing config checks to CMakeLists.txt	7 years ago
coverage	Fix /bin/bash shebangs	7 years ago
db	WritePrepared Txn: Fix DBIterator and add test	7 years ago
docs	fix Gemfile.lock nokogiri dependencies	7 years ago
env	Add a Close() method to DB to return status when closing a db	7 years ago
examples	Pinnableslice examples and blog post	7 years ago
hdfs	Revert "comment out unused parameters"	8 years ago
include/rocksdb	Update comments about default WALRecoveryMode	7 years ago
java	FreeBSD build support for RocksDB and RocksJava	7 years ago
memtable	fix gflags namespace	7 years ago
monitoring	fix ThreadStatus for bottom-pri compaction threads	7 years ago
options	DB::DumpSupportInfo should log all supported compression types	7 years ago
port	FreeBSD build support for RocksDB and RocksJava	7 years ago
table	Improve performance of long range scans with readahead	7 years ago
third-party	Enable MSVC W4 with a few exceptions. Fix warnings and bugs	7 years ago
tools	Add 5.10.fb to tools/check_format_compatible.sh	7 years ago
util	Add a Close() method to DB to return status when closing a db	7 years ago
utilities	Blob DB: dump blob_db_options.min_blob_size	7 years ago
.clang-format	A script that automatically reformat affected lines	11 years ago
.gitignore	Remove leftover references to phutil_module_cache	7 years ago
.travis.yml	CMake cross platform Java support and add JNI to travis	7 years ago
AUTHORS	Update RocksDB Authors File	7 years ago
CMakeLists.txt	CMake changes for CRC32 Optimization on PowerPC	7 years ago
CODE_OF_CONDUCT.md	Add Code of Conduct	7 years ago
CONTRIBUTING.md	Add Code of Conduct	7 years ago
COPYING	Add GPLv2 as an alternative license.	8 years ago
DEFAULT_OPTIONS_HISTORY.md	options.delayed_write_rate use the rate of rate_limiter by default.	8 years ago
DUMP_FORMAT.md	First version of rocksdb_dump and rocksdb_undump.	10 years ago
HISTORY.md	Improve performance of long range scans with readahead	7 years ago
INSTALL.md	FreeBSD build support for RocksDB and RocksJava	7 years ago
LANGUAGE-BINDINGS.md	Add Elixir to the list of language bindings	7 years ago
LICENSE.Apache	Change RocksDB License	8 years ago
LICENSE.leveldb	Add back the LevelDB license file	8 years ago
Makefile	Revert Snappy version upgrade	7 years ago
README.md	Add Jenkins for PPC64le build status badge	7 years ago
ROCKSDB_LITE.md	Optimistic Transactions	10 years ago
TARGETS	WritePrepared Txn: make buck tests parallel	7 years ago
USERS.md	Added ProfaneDB	7 years ago
Vagrantfile	Update Vagrant file (test internal phabricator workflow)	8 years ago
WINDOWS_PORT.md	Commit both PR and internal code review changes	10 years ago
appveyor.yml	Make Windows dep switches compatible with other builds	7 years ago
issue_template.md	Add a template for issues	7 years ago
src.mk	Refactor ReadBlockContents()	7 years ago
thirdparty.inc	Make Windows dep switches compatible with other builds	7 years ago

README.md

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

This code is a library that forms the core building block for a fast key value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it specially suitable for storing multiple terabytes of data in a single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/master/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.

Design discussions are conducted in https://www.facebook.com/groups/rocksdb.dev/