fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

gitbw95 4da34b97ee Set Read rate limiter priority dynamically and pass it to FS (#9996) ... Summary: ### Context: Background compactions and flush generate large reads and writes, and can be long running, especially for universal compaction. In some cases, this can impact foreground reads and writes by users. ### Solution User, Flush, and Compaction reads share some code path. For this task, we update the rate_limiter_priority in ReadOptions for code paths (e.g. FindTable (mainly in BlockBasedTable::Open()) and various iterators), and eventually update the rate_limiter_priority in IOOptions for FSRandomAccessFile. **This PR is for the Read path.** The **Read:** dynamic priority for different state are listed as follows: | State | Normal | Delayed | Stalled | | ----- | ------ | ------- | ------- | | Flush (verification read in BuildTable()) | IO_USER | IO_USER | IO_USER | | Compaction | IO_LOW | IO_USER | IO_USER | | User | User provided | User provided | User provided | We will respect the read_options that the user provided and will not set it. The only sst read for Flush is the verification read in BuildTable(). It claims to be "regard as user read". **Details** 1. Set read_options.rate_limiter_priority dynamically: - User: Do not update the read_options. Use the read_options that the user provided. - Compaction: Update read_options in CompactionJob::ProcessKeyValueCompaction(). - Flush: Update read_options in BuildTable(). 2. Pass the rate limiter priority to FSRandomAccessFile functions: - After calling the FindTable(), read_options is passed through GetTableReader(table_cache.cc), BlockBasedTableFactory::NewTableReader(block_based_table_factory.cc), and BlockBasedTable::Open(). The Open() needs some updates for the ReadOptions variable and the updates are also needed for the called functions, including PrefetchTail(), PrepareIOOptions(), ReadFooterFromFile(), ReadMetaIndexblock(), ReadPropertiesBlock(), PrefetchIndexAndFilterBlocks(), and ReadRangeDelBlock(). - In RandomAccessFileReader, the functions to be updated include Read(), MultiRead(), ReadAsync(), and Prefetch(). - Update the downstream functions of NewIndexIterator(), NewDataBlockIterator(), and BlockBasedTableIterator(). ### Test Plans Add unit tests. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9996 Reviewed By: anand1976 Differential Revision: D36452483 Pulled By: gitbw95 fbshipit-source-id: 60978204a4f849bb9261cb78d9bc1cb56d6008cf		3 years ago
..
blob	Remove own ToString() (#9955)	3 years ago
compaction	Set Read rate limiter priority dynamically and pass it to FS (#9996)	3 years ago
db_impl	Add table_properties_collector_factories override (#9995)	3 years ago
arena_wrapped_db_iter.cc	Remove own ToString() (#9955)	3 years ago
arena_wrapped_db_iter.h	Cleanup includes in dbformat.h (#8930)	3 years ago
builder.cc	Set Read rate limiter priority dynamically and pass it to FS (#9996)	3 years ago
builder.h	CompactionIterator sees consistent view of which keys are committed (#9830)	3 years ago
c.cc	Port the batched version of MultiGet() to RocksDB's C API (#9952)	3 years ago
c_test.c	Port the batched version of MultiGet() to RocksDB's C API (#9952)	3 years ago
column_family.cc	Use std::numeric_limits<> (#9954)	3 years ago
column_family.h	Meta-internal folly integration with F14FastMap (#9546)	3 years ago
column_family_test.cc	Remove own ToString() (#9955)	3 years ago
compact_files_test.cc	Remove own ToString() (#9955)	3 years ago
comparator_db_test.cc	Remove own ToString() (#9955)	3 years ago
convenience.cc	Specify largest_seqno in VerifyChecksum (#9919)	3 years ago
corruption_test.cc	Update WAL corruption test so that it fails without fix (#9942)	3 years ago
cuckoo_table_db_test.cc	Remove own ToString() (#9955)	3 years ago
db_basic_test.cc	Remove own ToString() (#9955)	3 years ago
db_block_cache_test.cc	Remove own ToString() (#9955)	3 years ago
db_bloom_filter_test.cc	Rewrite memory-charging feature's option API (#9926)	3 years ago
db_compaction_filter_test.cc	Remove own ToString() (#9955)	3 years ago
db_compaction_test.cc	Remove own ToString() (#9955)	3 years ago
db_dynamic_level_test.cc	Remove deprecated API AdvancedColumnFamilyOptions::soft_rate_limit/hard_rate_limit (#9452)	3 years ago
db_encryption_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_filesnapshot.cc	Use std::numeric_limits<> (#9954)	3 years ago
db_flush_test.cc	Use std::numeric_limits<> (#9954)	3 years ago
db_info_dumper.cc	Printing IO Error in DumpDBFileSummary (#9940)	3 years ago
db_info_dumper.h	Add a DB Session ID (#6959)	4 years ago
db_inplace_update_test.cc	fix a bug, c api, if enable inplace_update_support, and use create sn… (#9471)	3 years ago
db_io_failure_test.cc	Enable a few unit tests to use custom Env objects (#9087)	3 years ago
db_iter.cc	Remove dead code (#9825)	3 years ago
db_iter.h	Remove dead code (#9825)	3 years ago
db_iter_stress_test.cc	Remove own ToString() (#9955)	3 years ago
db_iter_test.cc	Remove own ToString() (#9955)	3 years ago
db_iterator_test.cc	Remove own ToString() (#9955)	3 years ago
db_kv_checksum_test.cc	Use std::numeric_limits<> (#9954)	3 years ago
db_log_iter_test.cc	Remove own ToString() (#9955)	3 years ago
db_logical_block_size_cache_test.cc	Attempt to deflake DBLogicalBlockSizeCacheTest.CreateColumnFamilies (#9516)	3 years ago
db_memtable_test.cc	Remove own ToString() (#9955)	3 years ago
db_merge_operand_test.cc	Fix GetMergeOperands() heap-use-after-free on flushed memtable (#9805)	3 years ago
db_merge_operator_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_options_test.cc	Remove own ToString() (#9955)	3 years ago
db_properties_test.cc	Remove own ToString() (#9955)	3 years ago
db_range_del_test.cc	Remove own ToString() (#9955)	3 years ago
db_rate_limiter_test.cc	Set Read rate limiter priority dynamically and pass it to FS (#9996)	3 years ago
db_secondary_test.cc	Remove own ToString() (#9955)	3 years ago
db_sst_test.cc	Rewrite memory-charging feature's option API (#9926)	3 years ago
db_statistics_test.cc	Bytes read stat for `VerifyChecksum()` and `VerifyFileChecksums()` APIs (#8741)	3 years ago
db_table_properties_test.cc	Remove own ToString() (#9955)	3 years ago
db_tailing_iter_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_test.cc	Revert "Bugfix/fix manual flush blocking bug (#9893)" (#9992)	3 years ago
db_test2.cc	Set Read rate limiter priority dynamically and pass it to FS (#9996)	3 years ago
db_test_util.cc	Rewrite memory-charging feature's option API (#9926)	3 years ago
db_test_util.h	Rewrite memory-charging feature's option API (#9926)	3 years ago
db_universal_compaction_test.cc	Remove own ToString() (#9955)	3 years ago
db_wal_test.cc	Remove own ToString() (#9955)	3 years ago
db_with_timestamp_basic_test.cc	Remove own ToString() (#9955)	3 years ago
db_with_timestamp_compaction_test.cc	Use the comparator from the sst file table properties in sst_dump_tool (#9491)	3 years ago
db_write_buffer_manager_test.cc	Enable a few unit tests to use custom Env objects (#9087)	3 years ago
db_write_test.cc	Remove own ToString() (#9955)	3 years ago
dbformat.cc	Track per-SST user-defined timestamp information in MANIFEST (#9092)	3 years ago
dbformat.h	Use std::numeric_limits<> (#9954)	3 years ago
dbformat_test.cc	Enable a few unit tests to use custom Env objects (#9087)	3 years ago
deletefile_test.cc	Remove own ToString() (#9955)	3 years ago
error_handler.cc	Expand auto recovery to background read errors (#9679)	3 years ago
error_handler.h	Expand auto recovery to background read errors (#9679)	3 years ago
error_handler_fs_test.cc	Remove own ToString() (#9955)	3 years ago
event_helpers.cc	Fix a race condition in WAL tracking causing DB open failure (#9715)	3 years ago
event_helpers.h	Add a listener callback for end of auto error recovery (#9244)	3 years ago
experimental.cc	Add manifest fix-up utility for file temperatures (#9683)	3 years ago
external_sst_file_basic_test.cc	Remove own ToString() (#9955)	3 years ago
external_sst_file_ingestion_job.cc	Log error message when LinkFile() is not supported when ingesting files (#10010)	3 years ago
external_sst_file_ingestion_job.h	New stable, fixed-length cache keys (#9126)	3 years ago
external_sst_file_test.cc	Remove own ToString() (#9955)	3 years ago
fault_injection_test.cc	Fix a bug causing duplicate trailing entries in WritableFile (buffered IO) (#9236)	3 years ago
file_indexer.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
file_indexer.h	Use std::numeric_limits<> (#9954)	3 years ago
file_indexer_test.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
filename_test.cc	fixing issue #8345 RocksDB does not work when using UNC network paths (#9384)	3 years ago
flush_job.cc	Set Write rate limiter priority dynamically and pass it to FS (#9988)	3 years ago
flush_job.h	Set Write rate limiter priority dynamically and pass it to FS (#9988)	3 years ago
flush_job_test.cc	Set Write rate limiter priority dynamically and pass it to FS (#9988)	3 years ago
flush_scheduler.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
flush_scheduler.h	Include C++ standard library headers instead of C compatibility headers (#8068)	4 years ago
forward_iterator.cc	Remove own ToString() (#9955)	3 years ago
forward_iterator.h	Fast path for detecting unchanged prefix_extractor (#9407)	3 years ago
forward_iterator_bench.cc	Remove using namespace (#9369)	3 years ago
history_trimming_iterator.h	Add OpenAndTrimHistory API to support trimming data with specified timestamp (#9410)	3 years ago
import_column_family_job.cc	Log error message when LinkFile() is not supported when ingesting files (#10010)	3 years ago
import_column_family_job.h	New stable, fixed-length cache keys (#9126)	3 years ago
import_column_family_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
internal_stats.cc	Remove own ToString() (#9955)	3 years ago
internal_stats.h	Expose the amount of garbage in live blob files as a dedicated DB property (#9835)	3 years ago
job_context.h	CompactionIterator sees consistent view of which keys are committed (#9830)	3 years ago
kv_checksum.h	fix compile errors in db/kv_checksum.h (#9173)	3 years ago
listener_test.cc	Remove own ToString() (#9955)	3 years ago
log_format.h	Add record to set WAL compression type if enabled (#9556)	3 years ago
log_reader.cc	Integrate WAL compression into log reader/writer. (#9642)	3 years ago
log_reader.h	Integrate WAL compression into log reader/writer. (#9642)	3 years ago
log_test.cc	Integrate WAL compression into log reader/writer. (#9642)	3 years ago
log_writer.cc	Integrate WAL compression into log reader/writer. (#9642)	3 years ago
log_writer.h	Integrate WAL compression into log reader/writer. (#9642)	3 years ago
logs_with_prep_tracker.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
logs_with_prep_tracker.h	Include C++ standard library headers instead of C compatibility headers (#8068)	4 years ago
lookup_key.h	Cleanup includes in dbformat.h (#8930)	3 years ago
malloc_stats.cc	Replace most typedef with using= (#8751)	3 years ago
malloc_stats.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
manual_compaction_test.cc	Remove using namespace (#9369)	3 years ago
memtable.cc	Use std::numeric_limits<> (#9954)	3 years ago
memtable.h	Meta-internal folly integration with F14FastMap (#9546)	3 years ago
memtable_list.cc	Encode min_log_number_to_keep and delete_wals_before in one version edit (#9766)	3 years ago
memtable_list.h	Fix various spelling errors still found in code (#9653)	3 years ago
memtable_list_test.cc	Remove own ToString() (#9955)	3 years ago
merge_context.h	Add Merge Operator support to WriteBatchWithIndex (#8135)	4 years ago
merge_helper.cc	Support readahead during compaction for blob files (#9187)	3 years ago
merge_helper.h	Support readahead during compaction for blob files (#9187)	3 years ago
merge_helper_test.cc	Support readahead during compaction for blob files (#9187)	3 years ago
merge_operator.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
merge_test.cc	Make the Env class Customizable (#9293)	3 years ago
obsolete_files_test.cc	Remove own ToString() (#9955)	3 years ago
options_file_test.cc	No elide constructors (#7798)	4 years ago
output_validator.cc	Cleanup includes in dbformat.h (#8930)	3 years ago
output_validator.h	Cleanup includes in dbformat.h (#8930)	3 years ago
perf_context_test.cc	Remove own ToString() (#9955)	3 years ago
periodic_work_scheduler.cc	Fix a timer crash caused by invalid memory management (#9656)	3 years ago
periodic_work_scheduler.h	Fix a timer crash caused by invalid memory management (#9656)	3 years ago
periodic_work_scheduler_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
pinned_iterators_manager.h	Replace most typedef with using= (#8751)	3 years ago
plain_table_db_test.cc	Remove own ToString() (#9955)	3 years ago
pre_release_callback.h	Fix and detect headers with missing dependencies (#8893)	3 years ago
prefix_test.cc	Remove own ToString() (#9955)	3 years ago
range_del_aggregator.cc	In ParseInternalKey(), include corrupt key info in Status (#7515)	4 years ago
range_del_aggregator.h	Fix some typos in comments (#8066)	4 years ago
range_del_aggregator_bench.cc	Cleanup multiple implementations of VectorIterator (#8901)	3 years ago
range_del_aggregator_test.cc	Cleanup multiple implementations of VectorIterator (#8901)	3 years ago
range_tombstone_fragmenter.cc	Added memtable garbage statistics (#8411)	3 years ago
range_tombstone_fragmenter.h	Added memtable garbage statistics (#8411)	3 years ago
range_tombstone_fragmenter_test.cc	Cleanup multiple implementations of VectorIterator (#8901)	3 years ago
read_callback.h	Fix and detect headers with missing dependencies (#8893)	3 years ago
repair.cc	Remove own ToString() (#9955)	3 years ago
repair_test.cc	Remove own ToString() (#9955)	3 years ago
snapshot_checker.h	Use STATIC_AVOID_DESTRUCTION for static objects with non-trivial destructors (#9958)	3 years ago
snapshot_impl.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
snapshot_impl.h	Fix and detect headers with missing dependencies (#8893)	3 years ago
table_cache.cc	Improve the precision of row entry charge in row_cache (#9337)	3 years ago
table_cache.h	Fast path for detecting unchanged prefix_extractor (#9407)	3 years ago
table_properties_collector.cc	Apply `sample_for_compression` to all block-based tables (#8105)	4 years ago
table_properties_collector.h	Track each SST's timestamp information as user properties (#9093)	3 years ago
table_properties_collector_test.cc	Improve / clean up meta block code & integrity (#9163)	3 years ago
transaction_log_impl.cc	Add checks to GetUpdatesSince (#9459)	3 years ago
transaction_log_impl.h	Add checks to GetUpdatesSince (#9459)	3 years ago
trim_history_scheduler.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
trim_history_scheduler.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
version_builder.cc	Use std::numeric_limits<> (#9954)	3 years ago
version_builder.h	Fast path for detecting unchanged prefix_extractor (#9407)	3 years ago
version_builder_test.cc	Remove own ToString() (#9955)	3 years ago
version_edit.cc	Remove own ToString() (#9955)	3 years ago
version_edit.h	Propagate errors from UpdateBoundaries (#9851)	3 years ago
version_edit_handler.cc	Fix a race condition in WAL tracking causing DB open failure (#9715)	3 years ago
version_edit_handler.h	Fixed manifest_dump issues when printing keys and values containing null characters (#8378)	3 years ago
version_edit_test.cc	Propagate errors from UpdateBoundaries (#9851)	3 years ago
version_set.cc	Remove own ToString() (#9955)	3 years ago
version_set.h	Use std::numeric_limits<> (#9954)	3 years ago
version_set_test.cc	Fix a race condition in WAL tracking causing DB open failure (#9715)	3 years ago
version_util.h	Add manifest fix-up utility for file temperatures (#9683)	3 years ago
wal_edit.cc	Always track WAL obsoletion (#7759)	4 years ago
wal_edit.h	Use std::numeric_limits<> (#9954)	3 years ago
wal_edit_test.cc	Always track WAL obsoletion (#7759)	4 years ago
wal_manager.cc	Remove own ToString() (#9955)	3 years ago
wal_manager.h	Add checks to GetUpdatesSince (#9459)	3 years ago
wal_manager_test.cc	Remove own ToString() (#9955)	3 years ago
write_batch.cc	Use std::numeric_limits<> (#9954)	3 years ago
write_batch_base.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
write_batch_internal.h	Support WBWI for keys having timestamps (#9603)	3 years ago
write_batch_test.cc	Remove own ToString() (#9955)	3 years ago
write_callback.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
write_callback_test.cc	Move slow valgrind tests behind -DROCKSDB_FULL_VALGRIND_RUN (#8475)	3 years ago
write_controller.cc	Revamp WriteController (#8064)	4 years ago
write_controller.h	Set Write rate limiter priority dynamically and pass it to FS (#9988)	3 years ago
write_controller_test.cc	Revamp WriteController (#8064)	4 years ago
write_thread.cc	Rate-limit automatic WAL flush after each user write (#9607)	3 years ago
write_thread.h	Rate-limit automatic WAL flush after each user write (#9607)	3 years ago