rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

History

gitbw95 05c678e135 Set Write rate limiter priority dynamically and pass it to FS (#9988 ) Summary: ### Context: Background compactions and flush generate large reads and writes, and can be long running, especially for universal compaction. In some cases, this can impact foreground reads and writes by users. From the RocksDB perspective, there can be two kinds of rate limiters, the internal (native) one and the external one. - The internal (native) rate limiter is introduced in [the wiki](https://github.com/facebook/rocksdb/wiki/Rate-Limiter). Currently, only IO_LOW and IO_HIGH are used and they are set statically. - For the external rate limiter, in FSWritableFile functions, IOOptions is open for end users to set and get rate_limiter_priority for their own rate limiter. Currently, RocksDB doesn’t pass the rate_limiter_priority through IOOptions to the file system. ### Solution During the User Read, Flush write, Compaction read/write, the WriteController is used to determine whether DB writes are stalled or slowed down. The rate limiter priority (Env::IOPriority) can be determined accordingly. We decided to always pass the priority in IOOptions. What the file system does with it should be a contract between the user and the file system. We would like to set the rate limiter priority at file level, since the Flush/Compaction job level may be too coarse with multiple files and block IO level is too granular. This PR is for the Write path. The Write: dynamic priority for different state are listed as follows: \| State \| Normal \| Delayed \| Stalled \| \| ----- \| ------ \| ------- \| ------- \| \| Flush \| IO_HIGH \| IO_USER \| IO_USER \| \| Compaction \| IO_LOW \| IO_USER \| IO_USER \| Flush and Compaction writes share the same call path through BlockBaseTableWriter, WritableFileWriter, and FSWritableFile. When a new FSWritableFile object is created, its io_priority_ can be set dynamically based on the state of the WriteController. In WritableFileWriter, before the call sites of FSWritableFile functions, WritableFileWriter::DecideRateLimiterPriority() determines the rate_limiter_priority. The options (IOOptions) argument of FSWritableFile functions will be updated with the rate_limiter_priority. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9988 Test Plan: Add unit tests. Reviewed By: anand1976 Differential Revision: D36395159 Pulled By: gitbw95 fbshipit-source-id: a7c82fc29759139a1a07ec46c37dbf7e753474cf		3 years ago
..
delete_scheduler.cc	Skip directory fsync for filesystem btrfs (#8903 )	3 years ago
delete_scheduler.h	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033 )	4 years ago
delete_scheduler_test.cc	Remove own ToString() (#9955 )	3 years ago
file_prefetch_buffer.cc	Fix bug in async_io path which reads incorrect length (#9916 )	3 years ago
file_prefetch_buffer.h	Use std::numeric_limits<> (#9954 )	3 years ago
file_util.cc	Avoid usage of ReopenWritableFile in db_stress (#9649 )	3 years ago
file_util.h	Avoid usage of ReopenWritableFile in db_stress (#9649 )	3 years ago
filename.cc	fixing issue #8345 RocksDB does not work when using UNC network paths (#9384 )	3 years ago
filename.h	Add (Live)FileStorageInfo API (#8968 )	3 years ago
line_file_reader.cc	Replace Status with IOStatus in the backupable_db (#8820 )	3 years ago
line_file_reader.h	Replace Status with IOStatus in the backupable_db (#8820 )	3 years ago
prefetch_test.cc	Add stats related to async prefetching (#9845 )	3 years ago
random_access_file_reader.cc	Update stats for Read and ReadAsync in random_access_file_reader for async prefetching (#9810 )	3 years ago
random_access_file_reader.h	Update stats for Read and ReadAsync in random_access_file_reader for async prefetching (#9810 )	3 years ago
random_access_file_reader_test.cc	Add rate limiter priority to ReadOptions (#9424 )	3 years ago
read_write_util.cc	Move old files to warm tier in FIFO compactions (#8310 )	3 years ago
read_write_util.h	Refactor: add LineFileReader and Status::MustCheck (#8026 )	4 years ago
readahead_file_info.h	Reuse internal auto readhead_size at each Level (expect L0) for Iterations (#9056 )	3 years ago
readahead_raf.cc	Make StringEnv, StringSink, StringSource use FS classes (#7786 )	4 years ago
readahead_raf.h	Make StringEnv, StringSink, StringSource use FS classes (#7786 )	4 years ago
sequence_file_reader.cc	Clarify some SequentialFileReader::Read logic (#10002 )	3 years ago
sequence_file_reader.h	Add listener API that notifies on IOError (#9177 )	3 years ago
sst_file_manager_impl.cc	Fix race condition in SstFileManagerImpl error recovery code (#9435 )	3 years ago
sst_file_manager_impl.h	Fix typo about file/sst_file_manager_impl.h (#9799 )	3 years ago
writable_file_writer.cc	Set Write rate limiter priority dynamically and pass it to FS (#9988 )	3 years ago
writable_file_writer.h	Set Write rate limiter priority dynamically and pass it to FS (#9988 )	3 years ago