Use SpecialEnv to speed up some slow BackupEngineRateLimitingTestWithParam (#9974)

Summary: **Context:** `BackupEngineRateLimitingTestWithParam.RateLimiting` and `BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup` involve creating backup and restoring of a big database with rate-limiting. Using the normal env with a normal clock requires real elapse of time (13702 - 19848 ms/per test). As suggested in https://github.com/facebook/rocksdb/pull/8722#discussion_r703698603, this PR is to speed it up with SpecialEnv (`time_elapse_only_sleep=true`) where its clock accepts fake elapse of time during rate-limiting (100 - 600 ms/per test) **Summary:** - Added TEST_ function to set clock of the default rate limiters in backup engine - Shrunk testdb by 10 times while keeping it big enough for testing - Renamed some test variables and reorganized some if-else branch for clarity without changing the test Pull Request resolved: https://github.com/facebook/rocksdb/pull/9974 Test Plan: - Run tests pre/post PR the same time to verify the tests are sped up by 90 - 95% `BackupEngineRateLimitingTestWithParam.RateLimiting` Pre: ``` [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/0 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/0 (11123 ms) [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/1 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/1 (9441 ms) [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/2 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/2 (11096 ms) [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/3 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/3 (9339 ms) [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/4 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/4 (11121 ms) [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/5 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/5 (9413 ms) [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/6 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/6 (11185 ms) [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/7 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/7 (9511 ms) [----------] 8 tests from RateLimiting/BackupEngineRateLimitingTestWithParam (82230 ms total) ``` Post: ``` [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/0 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/0 (395 ms) [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/1 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/1 (564 ms) [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/2 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/2 (358 ms) [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/3 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/3 (567 ms) [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/4 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/4 (173 ms) [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/5 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/5 (176 ms) [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/6 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/6 (191 ms) [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/7 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/7 (177 ms) [----------] 8 tests from RateLimiting/BackupEngineRateLimitingTestWithParam (2601 ms total) ``` `BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup` Pre: ``` [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/0 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/0 (7275 ms) [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/1 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/1 (3961 ms) [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/2 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/2 (7117 ms) [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/3 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/3 (3921 ms) [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/4 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/4 (19862 ms) [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/5 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/5 (10231 ms) [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/6 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/6 (19848 ms) [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/7 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/7 (10372 ms) [----------] 8 tests from RateLimiting/BackupEngineRateLimitingTestWithParam (82587 ms total) ``` Post: ``` [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/0 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/0 (157 ms) [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/1 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/1 (152 ms) [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/2 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/2 (160 ms) [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/3 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/3 (158 ms) [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/4 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/4 (155 ms) [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/5 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/5 (151 ms) [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/6 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/6 (146 ms) [ RUN ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/7 [ OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/7 (153 ms) [----------] 8 tests from RateLimiting/BackupEngineRateLimitingTestWithParam (1232 ms total) ``` Reviewed By: pdillinger Differential Revision: D36336345 Pulled By: hx235 fbshipit-source-id: 724c6ba745f95f56d4440a6d2f1e4512a2987589
3 years ago · e66e6d2faa
parent 204a42ca97
commit e66e6d2faa
3 changed files with 171 additions and 48 deletions
--- a/utilities/backup/backup_engine.cc
+++ b/utilities/backup/backup_engine.cc
@ -48,6 +48,7 @@
 #include "util/coding.h"
 #include "util/crc32c.h"
 #include "util/math.h"
+#include "util/rate_limiter.h"
 #include "util/string_util.h"
 #include "utilities/backup/backup_engine_impl.h"
 #include "utilities/checkpoint/checkpoint_impl.h"
@ -181,6 +182,61 @@ class BackupEngineImpl {
           BackupEngineOptions::kMaskNamingFlags;
  }

+  void TEST_SetDefaultRateLimitersClock(
+      const std::shared_ptr<SystemClock>& backup_rate_limiter_clock,
+      const std::shared_ptr<SystemClock>& restore_rate_limiter_clock) {
+    if (backup_rate_limiter_clock) {
+      assert(options_.backup_rate_limiter->IsInstanceOf(
+          GenericRateLimiter::kClassName()));
+      auto* backup_rate_limiter_options =
+          options_.backup_rate_limiter
+              ->GetOptions<GenericRateLimiter::GenericRateLimiterOptions>();
+
+      assert(backup_rate_limiter_options);
+      RateLimiter::Mode backup_rate_limiter_mode;
+      if (!options_.backup_rate_limiter->IsRateLimited(
+              RateLimiter::OpType::kRead)) {
+        backup_rate_limiter_mode = RateLimiter::Mode::kWritesOnly;
+      } else if (!options_.backup_rate_limiter->IsRateLimited(
+                     RateLimiter::OpType::kWrite)) {
+        backup_rate_limiter_mode = RateLimiter::Mode::kReadsOnly;
+      } else {
+        backup_rate_limiter_mode = RateLimiter::Mode::kAllIo;
+      }
+      options_.backup_rate_limiter.reset(new GenericRateLimiter(
+          backup_rate_limiter_options->max_bytes_per_sec,
+          backup_rate_limiter_options->refill_period_us,
+          backup_rate_limiter_options->fairness, backup_rate_limiter_mode,
+          backup_rate_limiter_clock, backup_rate_limiter_options->auto_tuned));
+    }
+
+    if (restore_rate_limiter_clock) {
+      assert(options_.restore_rate_limiter->IsInstanceOf(
+          GenericRateLimiter::kClassName()));
+      auto* restore_rate_limiter_options =
+          options_.restore_rate_limiter
+              ->GetOptions<GenericRateLimiter::GenericRateLimiterOptions>();
+      assert(restore_rate_limiter_options);
+
+      RateLimiter::Mode restore_rate_limiter_mode;
+      if (!options_.restore_rate_limiter->IsRateLimited(
+              RateLimiter::OpType::kRead)) {
+        restore_rate_limiter_mode = RateLimiter::Mode::kWritesOnly;
+      } else if (!options_.restore_rate_limiter->IsRateLimited(
+                     RateLimiter::OpType::kWrite)) {
+        restore_rate_limiter_mode = RateLimiter::Mode::kReadsOnly;
+      } else {
+        restore_rate_limiter_mode = RateLimiter::Mode::kAllIo;
+      }
+      options_.restore_rate_limiter.reset(new GenericRateLimiter(
+          restore_rate_limiter_options->max_bytes_per_sec,
+          restore_rate_limiter_options->refill_period_us,
+          restore_rate_limiter_options->fairness, restore_rate_limiter_mode,
+          restore_rate_limiter_clock,
+          restore_rate_limiter_options->auto_tuned));
+    }
+  }
+
 private:
  void DeleteChildren(const std::string& dir,
                      uint32_t file_type_filter = 0) const;
@ -942,6 +998,15 @@ class BackupEngineImplThreadSafe : public BackupEngine,
    impl_.schema_test_options_.reset(new TEST_BackupMetaSchemaOptions(options));
  }

+  // Not public API but used in testing
+  void TEST_SetDefaultRateLimitersClock(
+      const std::shared_ptr<SystemClock>& backup_rate_limiter_clock = nullptr,
+      const std::shared_ptr<SystemClock>& restore_rate_limiter_clock =
+          nullptr) {
+    impl_.TEST_SetDefaultRateLimitersClock(backup_rate_limiter_clock,
+                                           restore_rate_limiter_clock);
+  }
+
 private:
  mutable port::RWMutex mutex_;
  BackupEngineImpl impl_;
@ -3162,6 +3227,15 @@ void TEST_SetBackupMetaSchemaOptions(
  impl->TEST_SetBackupMetaSchemaOptions(options);
 }

+void TEST_SetDefaultRateLimitersClock(
+    BackupEngine* engine,
+    const std::shared_ptr<SystemClock>& backup_rate_limiter_clock,
+    const std::shared_ptr<SystemClock>& restore_rate_limiter_clock) {
+  BackupEngineImplThreadSafe* impl =
+      static_cast_with_check<BackupEngineImplThreadSafe>(engine);
+  impl->TEST_SetDefaultRateLimitersClock(backup_rate_limiter_clock,
+                                         restore_rate_limiter_clock);
+}
 }  // namespace ROCKSDB_NAMESPACE

 #endif  // ROCKSDB_LITE
--- a/utilities/backup/backup_engine_impl.h
+++ b/utilities/backup/backup_engine_impl.h
@ -25,5 +25,12 @@ struct TEST_BackupMetaSchemaOptions {
 void TEST_SetBackupMetaSchemaOptions(
    BackupEngine *engine, const TEST_BackupMetaSchemaOptions &options);

+// Modifies the BackupEngine(Impl) to use specified clocks for backup and
+// restore rate limiters created by default if not specified by users for
+// test speedup.
+void TEST_SetDefaultRateLimitersClock(
+    BackupEngine* engine,
+    const std::shared_ptr<SystemClock>& backup_rate_limiter_clock = nullptr,
+    const std::shared_ptr<SystemClock>& restore_rate_limiter_clock = nullptr);
 }  // namespace ROCKSDB_NAMESPACE
 #endif  // ROCKSDB_LITE
--- a/utilities/backup/backup_engine_test.cc
+++ b/utilities/backup/backup_engine_test.cc
@ -2598,80 +2598,119 @@ INSTANTIATE_TEST_CASE_P(

 TEST_P(BackupEngineRateLimitingTestWithParam, RateLimiting) {
  size_t const kMicrosPerSec = 1000 * 1000LL;
-
-  std::shared_ptr<RateLimiter> backupThrottler(NewGenericRateLimiter(1));
-  std::shared_ptr<RateLimiter> restoreThrottler(NewGenericRateLimiter(1));
-
-  bool makeThrottler = std::get<0>(GetParam());
-  if (makeThrottler) {
-    engine_options_->backup_rate_limiter = backupThrottler;
-    engine_options_->restore_rate_limiter = restoreThrottler;
-  }
-
+  const bool custom_rate_limiter = std::get<0>(GetParam());
  // iter 0 -- single threaded
  // iter 1 -- multi threaded
-  int iter = std::get<1>(GetParam());
+  const int iter = std::get<1>(GetParam());
  const std::pair<uint64_t, uint64_t> limit = std::get<2>(GetParam());
-
+  std::unique_ptr<Env> special_env(
+      new SpecialEnv(db_chroot_env_.get(), /*time_elapse_only_sleep*/ true));
  // destroy old data
-  DestroyDB(dbname_, Options());
-  if (makeThrottler) {
-    backupThrottler->SetBytesPerSecond(limit.first);
-    restoreThrottler->SetBytesPerSecond(limit.second);
+  Options options;
+  options.env = special_env.get();
+  DestroyDB(dbname_, options);
+
+  if (custom_rate_limiter) {
+    std::shared_ptr<RateLimiter> backup_rate_limiter =
+        std::make_shared<GenericRateLimiter>(
+            limit.first, 100 * 1000 /* refill_period_us */, 10 /* fairness */,
+            RateLimiter::Mode::kWritesOnly /* mode */,
+            special_env->GetSystemClock(), false /* auto_tuned */);
+    std::shared_ptr<RateLimiter> restore_rate_limiter =
+        std::make_shared<GenericRateLimiter>(
+            limit.second, 100 * 1000 /* refill_period_us */, 10 /* fairness */,
+            RateLimiter::Mode::kWritesOnly /* mode */,
+            special_env->GetSystemClock(), false /* auto_tuned */);
+    engine_options_->backup_rate_limiter = backup_rate_limiter;
+    engine_options_->restore_rate_limiter = restore_rate_limiter;
  } else {
    engine_options_->backup_rate_limit = limit.first;
    engine_options_->restore_rate_limit = limit.second;
  }
+
  engine_options_->max_background_operations = (iter == 0) ? 1 : 10;
  options_.compression = kNoCompression;
+
+  // Rate limiter uses `CondVar::TimedWait()`, which does not have access to the
+  // `Env` to advance its time according to the fake wait duration. The
+  // workaround is to install a callback that advance the `Env`'s mock time.
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
+      "GenericRateLimiter::Request:PostTimedWait", [&](void* arg) {
+        int64_t time_waited_us = *static_cast<int64_t*>(arg);
+        special_env->SleepForMicroseconds(static_cast<int>(time_waited_us));
+      });
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
+
  OpenDBAndBackupEngine(true);
-  size_t bytes_written = FillDB(db_.get(), 0, 100000);
+  TEST_SetDefaultRateLimitersClock(backup_engine_.get(),
+                                   special_env->GetSystemClock());

-  auto start_backup = db_chroot_env_->NowMicros();
+  size_t bytes_written = FillDB(db_.get(), 0, 10000);
+
+  auto start_backup = special_env->NowMicros();
  ASSERT_OK(backup_engine_->CreateNewBackup(db_.get(), false));
-  auto backup_time = db_chroot_env_->NowMicros() - start_backup;
+  auto backup_time = special_env->NowMicros() - start_backup;
+  CloseDBAndBackupEngine();
  auto rate_limited_backup_time = (bytes_written * kMicrosPerSec) / limit.first;
  ASSERT_GT(backup_time, 0.8 * rate_limited_backup_time);

-  CloseDBAndBackupEngine();
-
  OpenBackupEngine();
-  auto start_restore = db_chroot_env_->NowMicros();
+  TEST_SetDefaultRateLimitersClock(backup_engine_.get(),
+                                   special_env->GetSystemClock());
+
+  auto start_restore = special_env->NowMicros();
  ASSERT_OK(backup_engine_->RestoreDBFromLatestBackup(dbname_, dbname_));
-  auto restore_time = db_chroot_env_->NowMicros() - start_restore;
+  auto restore_time = special_env->NowMicros() - start_restore;
  CloseBackupEngine();
  auto rate_limited_restore_time =
      (bytes_written * kMicrosPerSec) / limit.second;
  ASSERT_GT(restore_time, 0.8 * rate_limited_restore_time);

-  AssertBackupConsistency(0, 0, 100000, 100010);
+  AssertBackupConsistency(0, 0, 10000, 10100);
+
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearCallBack(
+      "GenericRateLimiter::Request:PostTimedWait");
 }

 TEST_P(BackupEngineRateLimitingTestWithParam, RateLimitingVerifyBackup) {
  const std::size_t kMicrosPerSec = 1000 * 1000LL;
-  std::shared_ptr<RateLimiter> backupThrottler(NewGenericRateLimiter(
-      1, 100 * 1000 /* refill_period_us */, 10 /* fairness */,
-      RateLimiter::Mode::kAllIo /* mode */));
-
-  bool makeThrottler = std::get<0>(GetParam());
-  if (makeThrottler) {
-    engine_options_->backup_rate_limiter = backupThrottler;
-  }
-
-  bool is_single_threaded = std::get<1>(GetParam()) == 0 ? true : false;
-  engine_options_->max_background_operations = is_single_threaded ? 1 : 10;
-
+  const bool custom_rate_limiter = std::get<0>(GetParam());
  const std::uint64_t backup_rate_limiter_limit = std::get<2>(GetParam()).first;
-  if (makeThrottler) {
-    engine_options_->backup_rate_limiter->SetBytesPerSecond(
-        backup_rate_limiter_limit);
+  const bool is_single_threaded = std::get<1>(GetParam()) == 0 ? true : false;
+  std::unique_ptr<Env> special_env(
+      new SpecialEnv(db_chroot_env_.get(), /*time_elapse_only_sleep*/ true));
+
+  if (custom_rate_limiter) {
+    std::shared_ptr<RateLimiter> backup_rate_limiter =
+        std::make_shared<GenericRateLimiter>(
+            backup_rate_limiter_limit, 100 * 1000 /* refill_period_us */,
+            10 /* fairness */, RateLimiter::Mode::kAllIo /* mode */,
+            special_env->GetSystemClock(), false /* auto_tuned */);
+    engine_options_->backup_rate_limiter = backup_rate_limiter;
  } else {
    engine_options_->backup_rate_limit = backup_rate_limiter_limit;
  }

-  DestroyDB(dbname_, Options());
+  engine_options_->max_background_operations = is_single_threaded ? 1 : 10;
+
+  Options options;
+  options.env = special_env.get();
+  DestroyDB(dbname_, options);
+  // Rate limiter uses `CondVar::TimedWait()`, which does not have access to the
+  // `Env` to advance its time according to the fake wait duration. The
+  // workaround is to install a callback that advance the `Env`'s mock time.
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
+      "GenericRateLimiter::Request:PostTimedWait", [&](void* arg) {
+        int64_t time_waited_us = *static_cast<int64_t*>(arg);
+        special_env->SleepForMicroseconds(static_cast<int>(time_waited_us));
+      });
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
  OpenDBAndBackupEngine(true /* destroy_old_data */);
-  FillDB(db_.get(), 0, 100000);
+  TEST_SetDefaultRateLimitersClock(backup_engine_.get(),
+                                   special_env->GetSystemClock(), nullptr);
+  FillDB(db_.get(), 0, 10000);
+
  ASSERT_OK(backup_engine_->CreateNewBackup(db_.get(),
                                            false /* flush_before_backup */));

@ -2688,21 +2727,24 @@ TEST_P(BackupEngineRateLimitingTestWithParam, RateLimitingVerifyBackup) {
  for (BackupFileInfo backup_file_info : backup_info.file_details) {
    bytes_read_during_verify_backup += backup_file_info.size;
  }
-
-  auto start_verify_backup = db_chroot_env_->NowMicros();
+  auto start_verify_backup = special_env->NowMicros();
  ASSERT_OK(
      backup_engine_->VerifyBackup(backup_id, true /* verify_with_checksum */));
-  auto verify_backup_time = db_chroot_env_->NowMicros() - start_verify_backup;
+  auto verify_backup_time = special_env->NowMicros() - start_verify_backup;
  auto rate_limited_verify_backup_time =
      (bytes_read_during_verify_backup * kMicrosPerSec) /
      backup_rate_limiter_limit;
-
-  if (makeThrottler) {
+  if (custom_rate_limiter) {
    EXPECT_GE(verify_backup_time, 0.8 * rate_limited_verify_backup_time);
  }
+
  CloseDBAndBackupEngine();
-  AssertBackupConsistency(backup_id, 0, 100000, 100010);
-  DestroyDB(dbname_, Options());
+  AssertBackupConsistency(backup_id, 0, 10000, 10010);
+  DestroyDB(dbname_, options);
+
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearCallBack(
+      "GenericRateLimiter::Request:PostTimedWait");
 }

 TEST_P(BackupEngineRateLimitingTestWithParam, RateLimitingChargeReadInBackup) {