rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

History

gitbw95 4da34b97ee Set Read rate limiter priority dynamically and pass it to FS (#9996 ) Summary: ### Context: Background compactions and flush generate large reads and writes, and can be long running, especially for universal compaction. In some cases, this can impact foreground reads and writes by users. ### Solution User, Flush, and Compaction reads share some code path. For this task, we update the rate_limiter_priority in ReadOptions for code paths (e.g. FindTable (mainly in BlockBasedTable::Open()) and various iterators), and eventually update the rate_limiter_priority in IOOptions for FSRandomAccessFile. This PR is for the Read path. The Read: dynamic priority for different state are listed as follows: \| State \| Normal \| Delayed \| Stalled \| \| ----- \| ------ \| ------- \| ------- \| \| Flush (verification read in BuildTable()) \| IO_USER \| IO_USER \| IO_USER \| \| Compaction \| IO_LOW \| IO_USER \| IO_USER \| \| User \| User provided \| User provided \| User provided \| We will respect the read_options that the user provided and will not set it. The only sst read for Flush is the verification read in BuildTable(). It claims to be "regard as user read". Details 1. Set read_options.rate_limiter_priority dynamically: - User: Do not update the read_options. Use the read_options that the user provided. - Compaction: Update read_options in CompactionJob::ProcessKeyValueCompaction(). - Flush: Update read_options in BuildTable(). 2. Pass the rate limiter priority to FSRandomAccessFile functions: - After calling the FindTable(), read_options is passed through GetTableReader(table_cache.cc), BlockBasedTableFactory::NewTableReader(block_based_table_factory.cc), and BlockBasedTable::Open(). The Open() needs some updates for the ReadOptions variable and the updates are also needed for the called functions, including PrefetchTail(), PrepareIOOptions(), ReadFooterFromFile(), ReadMetaIndexblock(), ReadPropertiesBlock(), PrefetchIndexAndFilterBlocks(), and ReadRangeDelBlock(). - In RandomAccessFileReader, the functions to be updated include Read(), MultiRead(), ReadAsync(), and Prefetch(). - Update the downstream functions of NewIndexIterator(), NewDataBlockIterator(), and BlockBasedTableIterator(). ### Test Plans Add unit tests. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9996 Reviewed By: anand1976 Differential Revision: D36452483 Pulled By: gitbw95 fbshipit-source-id: 60978204a4f849bb9261cb78d9bc1cb56d6008cf		3 years ago
..
delete_scheduler.cc	Skip directory fsync for filesystem btrfs (#8903 )	3 years ago
delete_scheduler.h	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033 )	4 years ago
delete_scheduler_test.cc	Remove own ToString() (#9955 )	3 years ago
file_prefetch_buffer.cc	Fix bug in async_io path which reads incorrect length (#9916 )	3 years ago
file_prefetch_buffer.h	Use std::numeric_limits<> (#9954 )	3 years ago
file_util.cc	Avoid usage of ReopenWritableFile in db_stress (#9649 )	3 years ago
file_util.h	Set Read rate limiter priority dynamically and pass it to FS (#9996 )	3 years ago
filename.cc	fixing issue #8345 RocksDB does not work when using UNC network paths (#9384 )	3 years ago
filename.h	Add (Live)FileStorageInfo API (#8968 )	3 years ago
line_file_reader.cc	Replace Status with IOStatus in the backupable_db (#8820 )	4 years ago
line_file_reader.h	Replace Status with IOStatus in the backupable_db (#8820 )	4 years ago
prefetch_test.cc	Add stats related to async prefetching (#9845 )	3 years ago
random_access_file_reader.cc	Update stats for Read and ReadAsync in random_access_file_reader for async prefetching (#9810 )	3 years ago
random_access_file_reader.h	Set Read rate limiter priority dynamically and pass it to FS (#9996 )	3 years ago
random_access_file_reader_test.cc	Add rate limiter priority to ReadOptions (#9424 )	3 years ago
read_write_util.cc	Move old files to warm tier in FIFO compactions (#8310 )	4 years ago
read_write_util.h	Refactor: add LineFileReader and Status::MustCheck (#8026 )	4 years ago
readahead_file_info.h	Reuse internal auto readhead_size at each Level (expect L0) for Iterations (#9056 )	3 years ago
readahead_raf.cc	Make StringEnv, StringSink, StringSource use FS classes (#7786 )	4 years ago
readahead_raf.h	Make StringEnv, StringSink, StringSource use FS classes (#7786 )	4 years ago
sequence_file_reader.cc	Clarify some SequentialFileReader::Read logic (#10002 )	3 years ago
sequence_file_reader.h	Add listener API that notifies on IOError (#9177 )	3 years ago
sst_file_manager_impl.cc	Fix race condition in SstFileManagerImpl error recovery code (#9435 )	3 years ago
sst_file_manager_impl.h	Fix typo about file/sst_file_manager_impl.h (#9799 )	3 years ago
writable_file_writer.cc	Set Write rate limiter priority dynamically and pass it to FS (#9988 )	3 years ago
writable_file_writer.h	Set Write rate limiter priority dynamically and pass it to FS (#9988 )	3 years ago