fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Andrew Kryczka babe56ddba Add rate limiter priority to ReadOptions (#9424) ... Summary: Users can set the priority for file reads associated with their operation by setting `ReadOptions::rate_limiter_priority` to something other than `Env::IO_TOTAL`. Rate limiting `VerifyChecksum()` and `VerifyFileChecksums()` is the motivation for this PR, so it also includes benchmarks and minor bug fixes to get that working. `RandomAccessFileReader::Read()` already had support for rate limiting compaction reads. I changed that rate limiting to be non-specific to compaction, but rather performed according to the passed in `Env::IOPriority`. Now the compaction read rate limiting is supported by setting `rate_limiter_priority = Env::IO_LOW` on its `ReadOptions`. There is no default value for the new `Env::IOPriority` parameter to `RandomAccessFileReader::Read()`. That means this PR goes through all callers (in some cases multiple layers up the call stack) to find a `ReadOptions` to provide the priority. There are TODOs for cases I believe it would be good to let user control the priority some day (e.g., file footer reads), and no TODO in cases I believe it doesn't matter (e.g., trace file reads). The API doc only lists the missing cases where a file read associated with a provided `ReadOptions` cannot be rate limited. For cases like file ingestion checksum calculation, there is no API to provide `ReadOptions` or `Env::IOPriority`, so I didn't count that as missing. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9424 Test Plan: - new unit tests - new benchmarks on ~50MB database with 1MB/s read rate limit and 100ms refill interval; verified with strace reads are chunked (at 0.1MB per chunk) and spaced roughly 100ms apart. - setup command: `./db_bench -benchmarks=fillrandom,compact -db=/tmp/testdb -target_file_size_base=1048576 -disable_auto_compactions=true -file_checksum=true` - benchmarks command: `strace -ttfe pread64 ./db_bench -benchmarks=verifychecksum,verifyfilechecksums -use_existing_db=true -db=/tmp/testdb -rate_limiter_bytes_per_sec=1048576 -rate_limit_bg_reads=1 -rate_limit_user_ops=true -file_checksum=true` - crash test using IO_USER priority on non-validation reads with https://github.com/facebook/rocksdb/issues/9567 reverted: `python3 tools/db_crashtest.py blackbox --max_key=1000000 --write_buffer_size=524288 --target_file_size_base=524288 --level_compaction_dynamic_level_bytes=true --duration=3600 --rate_limit_bg_reads=true --rate_limit_user_ops=true --rate_limiter_bytes_per_sec=10485760 --interval=10` Reviewed By: hx235 Differential Revision: D33747386 Pulled By: ajkr fbshipit-source-id: a2d985e97912fba8c54763798e04f006ccc56e0c		3 years ago
..
CMakeLists.txt	Stress test for RocksDB transactions (#8936)	3 years ago
batched_ops_stress.cc	Revise APIs related to user-defined timestamp (#8946)	3 years ago
cf_consistency_stress.cc	Add rate limiter priority to ReadOptions (#9424)	3 years ago
db_stress.cc	…
db_stress_common.cc	db_stress: db_stress fails on custom filesystems. (#9352)	3 years ago
db_stress_common.h	Add rate limiter priority to ReadOptions (#9424)	3 years ago
db_stress_compaction_filter.h	…
db_stress_driver.cc	Fix a copy-paste bug related to background threads in db_stress (#9485)	3 years ago
db_stress_driver.h	…
db_stress_env_wrapper.h	Make the Env class Customizable (#9293)	3 years ago
db_stress_gflags.cc	Add rate limiter priority to ReadOptions (#9424)	3 years ago
db_stress_listener.cc	db_stress: db_stress fails on custom filesystems. (#9352)	3 years ago
db_stress_listener.h	db_stress: db_stress fails on custom filesystems. (#9352)	3 years ago
db_stress_shared_state.cc	…
db_stress_shared_state.h	Optimize db_stress setup phase (#9475)	3 years ago
db_stress_stat.cc	…
db_stress_stat.h	…
db_stress_table_properties_collector.h	…
db_stress_test_base.cc	Add rate limiter priority to ReadOptions (#9424)	3 years ago
db_stress_test_base.h	db_stress begin tracking expected state after verification (#9470)	3 years ago
db_stress_tool.cc	db_stress: db_stress fails on custom filesystems. (#9352)	3 years ago
expected_state.cc	Remove timestamp from key in expected state (#9525)	3 years ago
expected_state.h	db_stress verify with lost unsynced operations (#8966)	3 years ago
multi_ops_txns_stress.cc	Add rate limiter priority to ReadOptions (#9424)	3 years ago
multi_ops_txns_stress.h	Stress test for RocksDB transactions (#8936)	3 years ago
no_batched_ops_stress.cc	Add rate limiter priority to ReadOptions (#9424)	3 years ago