rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

Hui Xiao 3622cfa34a Add back io_uring stress test hack with DbStressFSWrapper for FS not supporting read async (#11404 ) Summary: Context/Summary: To better utilize `DbStressFSWrapper` for some assertion, https://github.com/facebook/rocksdb/pull/11288 removed an io_uring stress test hack for POSIX FS not supporting read async added in https://github.com/facebook/rocksdb/pull/11242 = It was removed based on the assumption that a later PR https://github.com/facebook/rocksdb/pull/11296 is sufficient to serve as an alternative workaround. But recent stress tests has shown the opposite, mostly because 11296 approach might be subjected to incompleteness when more `ReadOptions` are passed down as what https://github.com/facebook/rocksdb/pull/11288 has done. As a short-term solution to both work around POSIX FS constraint above and utilize `DbStressFSWrapper` for 11288 assertion, I proposed this PR. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11404 Test Plan: - Stress test ensures 11288's assertion is still effective in `DbStressFSWrapper` ``` ./db_stress --acquire_snapshot_one_in=10000 --adaptive_readahead=0 --allow_data_in_errors=True --async_io=1 --avoid_flush_during_recovery=1 --avoid_unnecessary_blocking_io=0 --backup_max_size=104857600 --backup_one_in=100000 --batch_protection_bytes_per_key=8 --block_size=16384 --bloom_bits=16 --bottommost_compression_type=disable --bytes_per_sync=0 --cache_index_and_filter_blocks=0 --cache_size=8388608 --cache_type=hyper_clock_cache --charge_compression_dictionary_building_buffer=0 --charge_file_metadata=1 --charge_filter_construction=1 --charge_table_reader=0 --checkpoint_one_in=1000000 --checksum_type=kxxHash64 --clear_column_family_one_in=0 --compact_files_one_in=1000000 --compact_range_one_in=1000000 --compaction_pri=1 --compaction_ttl=0 --compression_max_dict_buffer_bytes=32767 --compression_max_dict_bytes=16384 --compression_parallel_threads=1 --compression_type=lz4 --compression_use_zstd_dict_trainer=1 --compression_zstd_max_train_bytes=0 --continuous_verification_interval=0 --data_block_index_type=0 --db=$db --db_write_buffer_size=0 --delpercent=4 --delrangepercent=1 --destroy_db_initially=0 --detect_filter_construct_corruption=1 --disable_wal=0 --enable_compaction_filter=0 --enable_pipelined_write=1 --enable_thread_tracking=0 --expected_values_dir=$exp --fail_if_options_file_error=1 --fifo_allow_compaction=0 --file_checksum_impl=crc32c --flush_one_in=1000000 --format_version=4 --get_current_wal_file_one_in=0 --get_live_files_one_in=1000000 --get_property_one_in=1000000 --get_sorted_wal_files_one_in=0 --index_block_restart_interval=4 --index_type=0 --ingest_external_file_one_in=0 --initial_auto_readahead_size=16384 --iterpercent=10 --key_len_percent_dist=1,30,69 --kill_random_test=888887 --level_compaction_dynamic_level_bytes=0 --lock_wal_one_in=1000000 --log2_keys_per_lock=10 --long_running_snapshots=0 --manual_wal_flush_one_in=1000 --mark_for_compaction_one_file_in=10 --max_auto_readahead_size=16384 --max_background_compactions=20 --max_bytes_for_level_base=10485760 --max_key=25000000 --max_key_len=3 --max_manifest_file_size=1073741824 --max_write_batch_group_size_bytes=1048576 --max_write_buffer_number=3 --max_write_buffer_size_to_maintain=8388608 --memtable_prefix_bloom_size_ratio=0.1 --memtable_protection_bytes_per_key=4 --memtable_whole_key_filtering=0 --memtablerep=skip_list --min_write_buffer_number_to_merge=2 --mmap_read=0 --mock_direct_io=True --nooverwritepercent=1 --num_file_reads_for_auto_readahead=0 --open_files=100 --open_metadata_write_fault_one_in=0 --open_read_fault_one_in=0 --open_write_fault_one_in=0 --ops_per_thread=20000000 --optimize_filters_for_memory=0 --paranoid_file_checks=0 --partition_filters=0 --partition_pinning=0 --pause_background_one_in=1000000 --periodic_compaction_seconds=0 --prefix_size=1 --prefixpercent=5 --prepopulate_block_cache=1 --preserve_internal_time_seconds=36000 --progress_reports=0 --read_fault_one_in=32 --readahead_size=16384 --readpercent=45 --recycle_log_file_num=0 --reopen=20 --ribbon_starting_level=1 --secondary_cache_fault_one_in=32 --secondary_cache_uri=compressed_secondary_cache://capacity=8388608 --snapshot_hold_ops=100000 --sst_file_manager_bytes_per_sec=0 --sst_file_manager_bytes_per_truncate=0 --stats_dump_period_sec=10 --subcompactions=1 --sync=0 --sync_fault_injection=1 --target_file_size_base=2097152 --target_file_size_multiplier=2 --test_batches_snapshots=0 --top_level_index_pinning=2 --unpartitioned_pinning=3 --use_direct_io_for_flush_and_compaction=0 --use_direct_reads=1 --use_full_merge_v1=0 --use_get_entity=0 --use_merge=1 --use_multi_get_entity=0 --use_multiget=1 --use_put_entity_one_in=0 --user_timestamp_size=0 --value_size_mult=32 --verify_checksum=1 --verify_checksum_one_in=1000000 --verify_db_one_in=100000 --verify_sst_unique_id_in_manifest=1 --wal_bytes_per_sync=524288 --wal_compression=none --write_buffer_size=4194304 --write_dbid_to_manifest=1 --writepercent=35 ``` - Monitor future stress test to show `MultiGet error: Not implemented: ReadAsync` is gone Reviewed By: ltamasi Differential Revision: D45242280 Pulled By: hx235 fbshipit-source-id: 9823e3fbd4e9672efdd31478a2f2cbd68a98bdf5		2 years ago
.circleci	Revert enabling IO uring in db_stress (#11242 )	2 years ago
.github/workflows	ci: add GitHub token permissions for workflow (#10549 )	2 years ago
buckifier	Cleanup, improve, stress test LockWAL() (#11143 )	2 years ago
build_tools	Remove platform009 and default to platform010 (#11333 )	2 years ago
cache	fix bad implementation of ShardedCache::GetOccupancyCount (#11325 )	2 years ago
cmake	gcc-11 and cmake related cleanup (#9286 )	3 years ago
coverage	Remove platform009 and default to platform010 (#11333 )	2 years ago
db	Fix compression tests when snappy not available (#11396 )	2 years ago
db_stress_tool	Add back io_uring stress test hack with DbStressFSWrapper for FS not supporting read async (#11404 )	2 years ago
docs	Remove docs/Gemfile.lock and update github-pages version (#11173 )	2 years ago
env	Ignore async_io ReadOption if FileSystem doesn't support it (#11296 )	2 years ago
examples	Remove RocksDB LITE (#11147 )	2 years ago
file	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288 )	2 years ago
fuzz	Add some missing headers (#10519 )	2 years ago
include/rocksdb	Start version 8.3 (#11405 )	2 years ago
java	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288 )	2 years ago
logging	Disabling some IO error assertion in EnvLogger (#11314 )	2 years ago
memory	Remove RocksDB LITE (#11147 )	2 years ago
memtable	Add `SetAllowStall()` (#11335 )	2 years ago
microbench	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288 )	2 years ago
monitoring	Changes and enhancements to compression stats, thresholds (#11388 )	2 years ago
options	Changes and enhancements to compression stats, thresholds (#11388 )	2 years ago
plugin	Add initial CMake support to plugin (#9214 )	3 years ago
port	Ignore async_io ReadOption if FileSystem doesn't support it (#11296 )	2 years ago
table	Fix compression tests^2 (#11403 )	2 years ago
test_util	Fix compression tests when snappy not available (#11396 )	2 years ago
third-party	fix optimization-disabled test builds with platform010 (#11361 )	2 years ago
tools	Start version 8.3 (#11405 )	2 years ago
trace_replay	Fix error maybe-uninitialized #11100 (#11101 )	2 years ago
util	Changes and enhancements to compression stats, thresholds (#11388 )	2 years ago
utilities	Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288 )	2 years ago
.clang-format	A script that automatically reformat affected lines	11 years ago
.gitignore	Git ignore .clangd/ (#10817 )	2 years ago
.lgtm.yml	Create lgtm.yml for LGTM.com C/C++ analysis (#4058 )	6 years ago
.watchmanconfig	Added .watchmanconfig file to rocksdb repo (#5593 )	5 years ago
AUTHORS	Update RocksDB Authors File	7 years ago
CMakeLists.txt	New stat rocksdb.{cf\|db}-write-stall-stats exposed in a structural way (#11300 )	2 years ago
CODE_OF_CONDUCT.md	Adopt Contributor Covenant	5 years ago
CONTRIBUTING.md	Add Code of Conduct	7 years ago
COPYING	Add GPLv2 as an alternative license.	8 years ago
DEFAULT_OPTIONS_HISTORY.md	Add Options::DisableExtraChecks, clarify force_consistency_checks (#9363 )	3 years ago
DUMP_FORMAT.md	First version of rocksdb_dump and rocksdb_undump.	10 years ago
HISTORY.md	Start version 8.3 (#11405 )	2 years ago
INSTALL.md	Add instructions for installing googlebenchmark (#11282 )	2 years ago
LANGUAGE-BINDINGS.md	Add grocksdb in Go language bindings (#10498 )	2 years ago
LICENSE.Apache	Change RocksDB License	7 years ago
LICENSE.leveldb	Add back the LevelDB license file	7 years ago
Makefile	util/ribbon_test.cc: avoid ambiguous reversed operator error in c++20 (#11371 )	2 years ago
PLUGINS.md	Add pmem-rocksdb-plugin link in PLUGINs.md (#9934 )	3 years ago
README.md	Remove deprecated integration tests from README.md (#11354 )	2 years ago
TARGETS	New stat rocksdb.{cf\|db}-write-stall-stats exposed in a structural way (#11300 )	2 years ago
USERS.md	Add PaxosStore to USERS (#11357 )	2 years ago
Vagrantfile	Adding CentOS 7 Vagrantfile & build script	7 years ago
WINDOWS_PORT.md	Update branch name in WINDOWS_PORT.md (#8745 )	3 years ago
common.mk	Clean up variables for temporary directory (#9961 )	3 years ago
crash_test.mk	Allow a custom DB cleanup command to be passed to db_crashtest.py (#10883 )	2 years ago
issue_template.md	Add Google Group to Issue Template	5 years ago
rocksdb.pc.in	build: fix pkg-config file generation (#9953 )	2 years ago
src.mk	New stat rocksdb.{cf\|db}-write-stall-stats exposed in a structural way (#11300 )	2 years ago
thirdparty.inc	Fix build jemalloc api (#5470 )	5 years ago

README.md

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/main/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.

Questions and discussions are welcome on the RocksDB Developers Public Facebook group and email list on Google Groups.

License

RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.