NextGraph
/
rocksdb
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Use SpecialEnv to speed up some slow BackupEngineRateLimitingTestWithParam (#9974)

Browse Source 
Summary:
**Context:**
`BackupEngineRateLimitingTestWithParam.RateLimiting` and `BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup` involve creating backup and restoring of a big database with rate-limiting. Using the normal env with a normal clock requires real elapse of time (13702 - 19848 ms/per test). As suggested in https://github.com/facebook/rocksdb/pull/8722#discussion_r703698603, this PR is to speed it up with SpecialEnv (`time_elapse_only_sleep=true`) where its clock accepts fake elapse of time during rate-limiting (100 - 600 ms/per test)

**Summary:**
- Added TEST_ function to set clock of the default rate limiters in backup engine
- Shrunk testdb by 10 times while keeping it big enough for testing
- Renamed some test variables and reorganized some if-else branch for clarity without changing the test

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9974

Test Plan:
- Run tests pre/post PR the same time to verify the tests are sped up by 90 - 95%
`BackupEngineRateLimitingTestWithParam.RateLimiting`
Pre:
```
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/0
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/0 (11123 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/1
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/1 (9441 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/2
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/2 (11096 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/3
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/3 (9339 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/4
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/4 (11121 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/5
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/5 (9413 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/6
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/6 (11185 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/7
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/7 (9511 ms)
[----------] 8 tests from RateLimiting/BackupEngineRateLimitingTestWithParam (82230 ms total)
```
Post:
```
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/0
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/0 (395 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/1
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/1 (564 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/2
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/2 (358 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/3
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/3 (567 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/4
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/4 (173 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/5
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/5 (176 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/6
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/6 (191 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/7
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/7 (177 ms)
[----------] 8 tests from RateLimiting/BackupEngineRateLimitingTestWithParam (2601 ms total)
```
`BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup`
Pre:
```
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/0
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/0 (7275 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/1
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/1 (3961 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/2
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/2 (7117 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/3
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/3 (3921 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/4
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/4 (19862 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/5
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/5 (10231 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/6
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/6 (19848 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/7
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/7 (10372 ms)
[----------] 8 tests from RateLimiting/BackupEngineRateLimitingTestWithParam (82587 ms total)
```
Post:
```
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/0
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/0 (157 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/1
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/1 (152 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/2
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/2 (160 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/3
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/3 (158 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/4
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/4 (155 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/5
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/5 (151 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/6
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/6 (146 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/7
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/7 (153 ms)
[----------] 8 tests from RateLimiting/BackupEngineRateLimitingTestWithParam (1232 ms total)
```

Reviewed By: pdillinger

Differential Revision: D36336345

Pulled By: hx235

fbshipit-source-id: 724c6ba745f95f56d4440a6d2f1e4512a2987589
main
Hui Xiao 3 years ago committed by Facebook GitHub Bot
parent 204a42ca97
commit e66e6d2faa
3 changed files with 171 additions and 48 deletions
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Show Stats Download Patch File Download Diff File
  1. 74
      utilities/backup/backup_engine.cc
  2. 7
      utilities/backup/backup_engine_impl.h
  3. 138
      utilities/backup/backup_engine_test.cc

74
utilities/backup/backup_engine.cc
Unescape View File

@ -48,6 +48,7 @@
#include "util/coding.h" #include "util/coding.h"
#include "util/crc32c.h" #include "util/crc32c.h"
#include "util/math.h" #include "util/math.h"
#include "util/rate_limiter.h"
#include "util/string_util.h" #include "util/string_util.h"
#include "utilities/backup/backup_engine_impl.h" #include "utilities/backup/backup_engine_impl.h"
#include "utilities/checkpoint/checkpoint_impl.h" #include "utilities/checkpoint/checkpoint_impl.h"
@ -181,6 +182,61 @@ class BackupEngineImpl {
BackupEngineOptions::kMaskNamingFlags; BackupEngineOptions::kMaskNamingFlags;
} }
void TEST_SetDefaultRateLimitersClock(
const std::shared_ptr<SystemClock>& backup_rate_limiter_clock,
const std::shared_ptr<SystemClock>& restore_rate_limiter_clock) {
if (backup_rate_limiter_clock) {
assert(options_.backup_rate_limiter->IsInstanceOf(
GenericRateLimiter::kClassName()));
auto* backup_rate_limiter_options =
options_.backup_rate_limiter
->GetOptions<GenericRateLimiter::GenericRateLimiterOptions>();
assert(backup_rate_limiter_options);
RateLimiter::Mode backup_rate_limiter_mode;
if (!options_.backup_rate_limiter->IsRateLimited(
RateLimiter::OpType::kRead)) {
backup_rate_limiter_mode = RateLimiter::Mode::kWritesOnly;
} else if (!options_.backup_rate_limiter->IsRateLimited(
RateLimiter::OpType::kWrite)) {
backup_rate_limiter_mode = RateLimiter::Mode::kReadsOnly;
} else {
backup_rate_limiter_mode = RateLimiter::Mode::kAllIo;
}
options_.backup_rate_limiter.reset(new GenericRateLimiter(
backup_rate_limiter_options->max_bytes_per_sec,
backup_rate_limiter_options->refill_period_us,
backup_rate_limiter_options->fairness, backup_rate_limiter_mode,
backup_rate_limiter_clock, backup_rate_limiter_options->auto_tuned));
}
if (restore_rate_limiter_clock) {
assert(options_.restore_rate_limiter->IsInstanceOf(
GenericRateLimiter::kClassName()));
auto* restore_rate_limiter_options =
options_.restore_rate_limiter
->GetOptions<GenericRateLimiter::GenericRateLimiterOptions>();
assert(restore_rate_limiter_options);
RateLimiter::Mode restore_rate_limiter_mode;
if (!options_.restore_rate_limiter->IsRateLimited(
RateLimiter::OpType::kRead)) {
restore_rate_limiter_mode = RateLimiter::Mode::kWritesOnly;
} else if (!options_.restore_rate_limiter->IsRateLimited(
RateLimiter::OpType::kWrite)) {
restore_rate_limiter_mode = RateLimiter::Mode::kReadsOnly;
} else {
restore_rate_limiter_mode = RateLimiter::Mode::kAllIo;
}
options_.restore_rate_limiter.reset(new GenericRateLimiter(
restore_rate_limiter_options->max_bytes_per_sec,
restore_rate_limiter_options->refill_period_us,
restore_rate_limiter_options->fairness, restore_rate_limiter_mode,
restore_rate_limiter_clock,
restore_rate_limiter_options->auto_tuned));
}
}
private: private:
void DeleteChildren(const std::string& dir, void DeleteChildren(const std::string& dir,
uint32_t file_type_filter = 0) const; uint32_t file_type_filter = 0) const;
@ -942,6 +998,15 @@ class BackupEngineImplThreadSafe : public BackupEngine,
impl_.schema_test_options_.reset(new TEST_BackupMetaSchemaOptions(options)); impl_.schema_test_options_.reset(new TEST_BackupMetaSchemaOptions(options));
} }
// Not public API but used in testing
void TEST_SetDefaultRateLimitersClock(
const std::shared_ptr<SystemClock>& backup_rate_limiter_clock = nullptr,
const std::shared_ptr<SystemClock>& restore_rate_limiter_clock =
nullptr) {
impl_.TEST_SetDefaultRateLimitersClock(backup_rate_limiter_clock,
restore_rate_limiter_clock);
}
private: private:
mutable port::RWMutex mutex_; mutable port::RWMutex mutex_;
BackupEngineImpl impl_; BackupEngineImpl impl_;
@ -3162,6 +3227,15 @@ void TEST_SetBackupMetaSchemaOptions(
impl->TEST_SetBackupMetaSchemaOptions(options); impl->TEST_SetBackupMetaSchemaOptions(options);
} }
void TEST_SetDefaultRateLimitersClock(
BackupEngine* engine,
const std::shared_ptr<SystemClock>& backup_rate_limiter_clock,
const std::shared_ptr<SystemClock>& restore_rate_limiter_clock) {
BackupEngineImplThreadSafe* impl =
static_cast_with_check<BackupEngineImplThreadSafe>(engine);
impl->TEST_SetDefaultRateLimitersClock(backup_rate_limiter_clock,
restore_rate_limiter_clock);
}
} // namespace ROCKSDB_NAMESPACE } // namespace ROCKSDB_NAMESPACE
#endif // ROCKSDB_LITE #endif // ROCKSDB_LITE

7
utilities/backup/backup_engine_impl.h
Unescape View File

@ -25,5 +25,12 @@ struct TEST_BackupMetaSchemaOptions {
void TEST_SetBackupMetaSchemaOptions( void TEST_SetBackupMetaSchemaOptions(
BackupEngine *engine, const TEST_BackupMetaSchemaOptions &options); BackupEngine *engine, const TEST_BackupMetaSchemaOptions &options);
// Modifies the BackupEngine(Impl) to use specified clocks for backup and
// restore rate limiters created by default if not specified by users for
// test speedup.
void TEST_SetDefaultRateLimitersClock(
BackupEngine* engine,
const std::shared_ptr<SystemClock>& backup_rate_limiter_clock = nullptr,
const std::shared_ptr<SystemClock>& restore_rate_limiter_clock = nullptr);
} // namespace ROCKSDB_NAMESPACE } // namespace ROCKSDB_NAMESPACE
#endif // ROCKSDB_LITE #endif // ROCKSDB_LITE

138
utilities/backup/backup_engine_test.cc
Unescape View File

@ -2598,80 +2598,119 @@ INSTANTIATE_TEST_CASE_P(
TEST_P(BackupEngineRateLimitingTestWithParam, RateLimiting) { TEST_P(BackupEngineRateLimitingTestWithParam, RateLimiting) {
size_t const kMicrosPerSec = 1000 * 1000LL; size_t const kMicrosPerSec = 1000 * 1000LL;
const bool custom_rate_limiter = std::get<0>(GetParam());
std::shared_ptr<RateLimiter> backupThrottler(NewGenericRateLimiter(1));
std::shared_ptr<RateLimiter> restoreThrottler(NewGenericRateLimiter(1));
bool makeThrottler = std::get<0>(GetParam());
if (makeThrottler) {
engine_options_->backup_rate_limiter = backupThrottler;
engine_options_->restore_rate_limiter = restoreThrottler;
}
// iter 0 -- single threaded // iter 0 -- single threaded
// iter 1 -- multi threaded // iter 1 -- multi threaded
int iter = std::get<1>(GetParam()); const int iter = std::get<1>(GetParam());
const std::pair<uint64_t, uint64_t> limit = std::get<2>(GetParam()); const std::pair<uint64_t, uint64_t> limit = std::get<2>(GetParam());
std::unique_ptr<Env> special_env(
new SpecialEnv(db_chroot_env_.get(), /*time_elapse_only_sleep*/ true));
// destroy old data // destroy old data
DestroyDB(dbname_, Options()); Options options;
if (makeThrottler) { options.env = special_env.get();
backupThrottler->SetBytesPerSecond(limit.first); DestroyDB(dbname_, options);
restoreThrottler->SetBytesPerSecond(limit.second);
if (custom_rate_limiter) {
std::shared_ptr<RateLimiter> backup_rate_limiter =
std::make_shared<GenericRateLimiter>(
limit.first, 100 * 1000 /* refill_period_us */, 10 /* fairness */,
RateLimiter::Mode::kWritesOnly /* mode */,
special_env->GetSystemClock(), false /* auto_tuned */);
std::shared_ptr<RateLimiter> restore_rate_limiter =
std::make_shared<GenericRateLimiter>(
limit.second, 100 * 1000 /* refill_period_us */, 10 /* fairness */,
RateLimiter::Mode::kWritesOnly /* mode */,
special_env->GetSystemClock(), false /* auto_tuned */);
engine_options_->backup_rate_limiter = backup_rate_limiter;
engine_options_->restore_rate_limiter = restore_rate_limiter;
} else { } else {
engine_options_->backup_rate_limit = limit.first; engine_options_->backup_rate_limit = limit.first;
engine_options_->restore_rate_limit = limit.second; engine_options_->restore_rate_limit = limit.second;
} }
engine_options_->max_background_operations = (iter == 0) ? 1 : 10; engine_options_->max_background_operations = (iter == 0) ? 1 : 10;
options_.compression = kNoCompression; options_.compression = kNoCompression;
// Rate limiter uses `CondVar::TimedWait()`, which does not have access to the
// `Env` to advance its time according to the fake wait duration. The
// workaround is to install a callback that advance the `Env`'s mock time.
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"GenericRateLimiter::Request:PostTimedWait", [&](void* arg) {
int64_t time_waited_us = *static_cast<int64_t*>(arg);
special_env->SleepForMicroseconds(static_cast<int>(time_waited_us));
});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
OpenDBAndBackupEngine(true); OpenDBAndBackupEngine(true);
size_t bytes_written = FillDB(db_.get(), 0, 100000); TEST_SetDefaultRateLimitersClock(backup_engine_.get(),
special_env->GetSystemClock());
auto start_backup = db_chroot_env_->NowMicros(); size_t bytes_written = FillDB(db_.get(), 0, 10000);
auto start_backup = special_env->NowMicros();
ASSERT_OK(backup_engine_->CreateNewBackup(db_.get(), false)); ASSERT_OK(backup_engine_->CreateNewBackup(db_.get(), false));
auto backup_time = db_chroot_env_->NowMicros() - start_backup; auto backup_time = special_env->NowMicros() - start_backup;
CloseDBAndBackupEngine();
auto rate_limited_backup_time = (bytes_written * kMicrosPerSec) / limit.first; auto rate_limited_backup_time = (bytes_written * kMicrosPerSec) / limit.first;
ASSERT_GT(backup_time, 0.8 * rate_limited_backup_time); ASSERT_GT(backup_time, 0.8 * rate_limited_backup_time);
CloseDBAndBackupEngine();
OpenBackupEngine(); OpenBackupEngine();
auto start_restore = db_chroot_env_->NowMicros(); TEST_SetDefaultRateLimitersClock(backup_engine_.get(),
special_env->GetSystemClock());
auto start_restore = special_env->NowMicros();
ASSERT_OK(backup_engine_->RestoreDBFromLatestBackup(dbname_, dbname_)); ASSERT_OK(backup_engine_->RestoreDBFromLatestBackup(dbname_, dbname_));
auto restore_time = db_chroot_env_->NowMicros() - start_restore; auto restore_time = special_env->NowMicros() - start_restore;
CloseBackupEngine(); CloseBackupEngine();
auto rate_limited_restore_time = auto rate_limited_restore_time =
(bytes_written * kMicrosPerSec) / limit.second; (bytes_written * kMicrosPerSec) / limit.second;
ASSERT_GT(restore_time, 0.8 * rate_limited_restore_time); ASSERT_GT(restore_time, 0.8 * rate_limited_restore_time);
AssertBackupConsistency(0, 0, 100000, 100010); AssertBackupConsistency(0, 0, 10000, 10100);
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearCallBack(
"GenericRateLimiter::Request:PostTimedWait");
} }
TEST_P(BackupEngineRateLimitingTestWithParam, RateLimitingVerifyBackup) { TEST_P(BackupEngineRateLimitingTestWithParam, RateLimitingVerifyBackup) {
const std::size_t kMicrosPerSec = 1000 * 1000LL; const std::size_t kMicrosPerSec = 1000 * 1000LL;
std::shared_ptr<RateLimiter> backupThrottler(NewGenericRateLimiter( const bool custom_rate_limiter = std::get<0>(GetParam());
1, 100 * 1000 /* refill_period_us */, 10 /* fairness */,
RateLimiter::Mode::kAllIo /* mode */));
bool makeThrottler = std::get<0>(GetParam());
if (makeThrottler) {
engine_options_->backup_rate_limiter = backupThrottler;
}
bool is_single_threaded = std::get<1>(GetParam()) == 0 ? true : false;
engine_options_->max_background_operations = is_single_threaded ? 1 : 10;
const std::uint64_t backup_rate_limiter_limit = std::get<2>(GetParam()).first; const std::uint64_t backup_rate_limiter_limit = std::get<2>(GetParam()).first;
if (makeThrottler) { const bool is_single_threaded = std::get<1>(GetParam()) == 0 ? true : false;
engine_options_->backup_rate_limiter->SetBytesPerSecond( std::unique_ptr<Env> special_env(
backup_rate_limiter_limit); new SpecialEnv(db_chroot_env_.get(), /*time_elapse_only_sleep*/ true));
if (custom_rate_limiter) {
std::shared_ptr<RateLimiter> backup_rate_limiter =
std::make_shared<GenericRateLimiter>(
backup_rate_limiter_limit, 100 * 1000 /* refill_period_us */,
10 /* fairness */, RateLimiter::Mode::kAllIo /* mode */,
special_env->GetSystemClock(), false /* auto_tuned */);
engine_options_->backup_rate_limiter = backup_rate_limiter;
} else { } else {
engine_options_->backup_rate_limit = backup_rate_limiter_limit; engine_options_->backup_rate_limit = backup_rate_limiter_limit;
} }
DestroyDB(dbname_, Options()); engine_options_->max_background_operations = is_single_threaded ? 1 : 10;
Options options;
options.env = special_env.get();
DestroyDB(dbname_, options);
// Rate limiter uses `CondVar::TimedWait()`, which does not have access to the
// `Env` to advance its time according to the fake wait duration. The
// workaround is to install a callback that advance the `Env`'s mock time.
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"GenericRateLimiter::Request:PostTimedWait", [&](void* arg) {
int64_t time_waited_us = *static_cast<int64_t*>(arg);
special_env->SleepForMicroseconds(static_cast<int>(time_waited_us));
});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
OpenDBAndBackupEngine(true /* destroy_old_data */); OpenDBAndBackupEngine(true /* destroy_old_data */);
FillDB(db_.get(), 0, 100000); TEST_SetDefaultRateLimitersClock(backup_engine_.get(),
special_env->GetSystemClock(), nullptr);
FillDB(db_.get(), 0, 10000);
ASSERT_OK(backup_engine_->CreateNewBackup(db_.get(), ASSERT_OK(backup_engine_->CreateNewBackup(db_.get(),
false /* flush_before_backup */)); false /* flush_before_backup */));
@ -2688,21 +2727,24 @@ TEST_P(BackupEngineRateLimitingTestWithParam, RateLimitingVerifyBackup) {
for (BackupFileInfo backup_file_info : backup_info.file_details) { for (BackupFileInfo backup_file_info : backup_info.file_details) {
bytes_read_during_verify_backup += backup_file_info.size; bytes_read_during_verify_backup += backup_file_info.size;
} }
auto start_verify_backup = special_env->NowMicros();
auto start_verify_backup = db_chroot_env_->NowMicros();
ASSERT_OK( ASSERT_OK(
backup_engine_->VerifyBackup(backup_id, true /* verify_with_checksum */)); backup_engine_->VerifyBackup(backup_id, true /* verify_with_checksum */));
auto verify_backup_time = db_chroot_env_->NowMicros() - start_verify_backup; auto verify_backup_time = special_env->NowMicros() - start_verify_backup;
auto rate_limited_verify_backup_time = auto rate_limited_verify_backup_time =
(bytes_read_during_verify_backup * kMicrosPerSec) / (bytes_read_during_verify_backup * kMicrosPerSec) /
backup_rate_limiter_limit; backup_rate_limiter_limit;
if (custom_rate_limiter) {
if (makeThrottler) {
EXPECT_GE(verify_backup_time, 0.8 * rate_limited_verify_backup_time); EXPECT_GE(verify_backup_time, 0.8 * rate_limited_verify_backup_time);
} }
CloseDBAndBackupEngine(); CloseDBAndBackupEngine();
AssertBackupConsistency(backup_id, 0, 100000, 100010); AssertBackupConsistency(backup_id, 0, 10000, 10010);
DestroyDB(dbname_, Options()); DestroyDB(dbname_, options);
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearCallBack(
"GenericRateLimiter::Request:PostTimedWait");
} }
TEST_P(BackupEngineRateLimitingTestWithParam, RateLimitingChargeReadInBackup) { TEST_P(BackupEngineRateLimitingTestWithParam, RateLimitingChargeReadInBackup) {

Write Preview
Loading…
Cancel
Save