fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Changyu Bi 8515bd50c9 Support read rate-limiting in SequentialFileReader (#9973) ... Summary: Added rate limiter and read rate-limiting support to SequentialFileReader. I've updated call sites to SequentialFileReader::Read with appropriate IO priority (or left a TODO and specified IO_TOTAL for now). The PR is separated into four commits: the first one added the rate-limiting support, but with some fixes in the unit test since the number of request bytes from rate limiter in SequentialFileReader are not accurate (there is overcharge at EOF). The second commit fixed this by allowing SequentialFileReader to check file size and determine how many bytes are left in the file to read. The third commit added benchmark related code. The fourth commit moved the logic of using file size to avoid overcharging the rate limiter into backup engine (the main user of SequentialFileReader). Pull Request resolved: https://github.com/facebook/rocksdb/pull/9973 Test Plan: - `make check`, backup_engine_test covers usage of SequentialFileReader with rate limiter. - Run db_bench to check if rate limiting is throttling as expected: Verified that reads and writes are together throttled at 2MB/s, and at 0.2MB chunks that are 100ms apart. - Set up: `./db_bench --benchmarks=fillrandom -db=/dev/shm/test_rocksdb` - Benchmark: ``` strace -ttfe read,write ./db_bench --benchmarks=backup -db=/dev/shm/test_rocksdb --backup_rate_limit=2097152 --use_existing_db strace -ttfe read,write ./db_bench --benchmarks=restore -db=/dev/shm/test_rocksdb --restore_rate_limit=2097152 --use_existing_db ``` - db bench on backup and restore to ensure no performance regression. - backup (avg over 50 runs): pre-change: 1.90443e+06 micros/op; post-change: 1.8993e+06 micros/op (improve by 0.2%) - restore (avg over 50 runs): pre-change: 1.79105e+06 micros/op; post-change: 1.78192e+06 micros/op (improve by 0.5%) ``` # Set up ./db_bench --benchmarks=fillrandom -db=/tmp/test_rocksdb -num=10000000 # benchmark TEST_TMPDIR=/tmp/test_rocksdb NUM_RUN=50 for ((j=0;j<$NUM_RUN;j++)) do ./db_bench -db=$TEST_TMPDIR -num=10000000 -benchmarks=backup -use_existing_db | egrep 'backup' # Restore #./db_bench -db=$TEST_TMPDIR -num=10000000 -benchmarks=restore -use_existing_db done > rate_limit.txt && awk -v NUM_RUN=$NUM_RUN '{sum+=$3;sum_sqrt+=$3^2}END{print sum/NUM_RUN, sqrt(sum_sqrt/NUM_RUN-(sum/NUM_RUN)^2)}' rate_limit.txt >> rate_limit_2.txt ``` Reviewed By: hx235 Differential Revision: D36327418 Pulled By: cbi42 fbshipit-source-id: e75d4307cff815945482df5ba630c1e88d064691		2 years ago
..
cf_options.cc	Support using ZDICT_finalizeDictionary to generate zstd dictionary (#9857)	2 years ago
cf_options.h	`compression_per_level` should be used for flush and changeable (#9658)	3 years ago
configurable.cc	Option type info functions (#9411)	2 years ago
configurable_helper.h	Fix some minor issues in the Customizable infrastructure (#8566)	3 years ago
configurable_test.cc	Remove own ToString() (#9955)	2 years ago
configurable_test.h	Move RegisterOptions into the Configurable API (#8223)	3 years ago
customizable.cc	Option type info functions (#9411)	2 years ago
customizable_test.cc	Added GetFactoryCount/Names/Types to ObjectRegistry (#9358)	2 years ago
db_options.cc	Track SST unique id in MANIFEST and verify (#9990)	2 years ago
db_options.h	Track SST unique id in MANIFEST and verify (#9990)	2 years ago
options.cc	Support using ZDICT_finalizeDictionary to generate zstd dictionary (#9857)	2 years ago
options_helper.cc	Fix a bug of not setting enforce_single_del_contracts (#10027)	2 years ago
options_helper.h	New backup meta schema, with file temperatures (#9660)	3 years ago
options_parser.cc	Support read rate-limiting in SequentialFileReader (#9973)	2 years ago
options_parser.h	Bring the Configurable options together (#5753)	4 years ago
options_settable_test.cc	Fix a bug of not setting enforce_single_del_contracts (#10027)	2 years ago
options_test.cc	Support using ZDICT_finalizeDictionary to generate zstd dictionary (#9857)	2 years ago