Crash test to make kill decision for every kill point

Summary:
In crash test, when coming to each kill point, we start a random class using seed as current second. With this approach, for every second, the random number used is the same. However, in each second, there are multiple kill points with different frequency. It makes it hard to reason about chance of kill point to trigger. With this commit, we use thread local random seed to generate the random number, so that it will take different values per second, hoping it makes chances of killing much easier to reason about.

Also significantly reduce the kill odd to make sure time before kiling is similar as before.

Test Plan: Run white box crash test and see the killing happens as expected and the run time time before killing reasonable.

Reviewers: kradhakrishnan, IslamAbdelRahman, rven, yhchiang, andrewkr, anthony

Reviewed By: anthony

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D52971

tools/db_crashtest.py

@@ -236,7 +236,7 @@ def whitebox_crash_main(args):
 
     total_check_mode = 4
     check_mode = 0
-    kill_random_test = 97
+    kill_random_test = 888887
     kill_mode = 0
 
     while time.time() < exit_time:
@@ -255,13 +255,13 @@ def whitebox_crash_main(args):
                 })
             elif kill_mode == 1:
                 additional_opts.update({
-                    "kill_random_test": (kill_random_test / 2 + 1),
+                    "kill_random_test": (kill_random_test / 10 + 1),
                     "kill_prefix_blacklist": "WritableFileWriter::Append,"
                     + "WritableFileWriter::WriteBuffered",
                 })
             elif kill_mode == 2:
                 additional_opts.update({
-                    "kill_random_test": (kill_random_test / 4 + 1),
+                    "kill_random_test": (kill_random_test / 5000 + 1),
                     "kill_prefix_blacklist": "WritableFileWriter::Append,"
                     "WritableFileWriter::WriteBuffered,"
                     "PosixMmapFile::Allocate,WritableFileWriter::Flush",

util/sync_point.cc

@@ -21,16 +21,14 @@ void TestKillRandom(std::string kill_point, int odds,
     }
   }
 
-  time_t curtime = time(nullptr);
-  Random r((uint32_t)curtime);
-
   assert(odds > 0);
   if (odds % 7 == 0) {
-    // class Rarndom uses multiplier 16807, which is 7^5. If odds are
-    // multiplier of 7, the first random value might have limited values.
+    // class Random uses multiplier 16807, which is 7^5. If odds are
+    // multiplier of 7, there might be limited values generated.
     odds++;
   }
-  bool crash = r.OneIn(odds);
+  auto* r = Random::GetTLSInstance();
+  bool crash = r->OneIn(odds);
   if (crash) {
     port::Crash(srcfile, srcline);
   }