fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
160 lines
7.2 KiB
160 lines
7.2 KiB
# Fuzzing RocksDB
|
|
|
|
## Overview
|
|
|
|
This directory contains [fuzz tests](https://en.wikipedia.org/wiki/Fuzzing) for RocksDB.
|
|
RocksDB testing infrastructure currently includes unit tests and [stress tests](https://github.com/facebook/rocksdb/wiki/Stress-test),
|
|
we hope fuzz testing can catch more bugs.
|
|
|
|
## Prerequisite
|
|
|
|
We use [LLVM libFuzzer](http://llvm.org/docs/LibFuzzer.html) as the fuzzying engine,
|
|
so make sure you have [clang](https://clang.llvm.org/get_started.html) as your compiler.
|
|
|
|
Some tests rely on [structure aware fuzzing](https://github.com/google/fuzzing/blob/master/docs/structure-aware-fuzzing.md).
|
|
We use [protobuf](https://developers.google.com/protocol-buffers) to define structured input to the fuzzer,
|
|
and use [libprotobuf-mutator](https://github.com/google/libprotobuf-mutator) as the custom libFuzzer mutator.
|
|
So make sure you have protobuf and libprotobuf-mutator installed, and make sure `pkg-config` can find them.
|
|
|
|
## Example
|
|
|
|
This example shows you how to do structure aware fuzzing to `rocksdb::SstFileWriter`.
|
|
|
|
After walking through the steps to create the fuzzer, we'll introduce a bug into `rocksdb::SstFileWriter::Put`,
|
|
then show that the fuzzer can catch the bug.
|
|
|
|
### Design the test
|
|
|
|
We want the fuzzing engine to automatically generate a list of database operations,
|
|
then we apply these operations to `SstFileWriter` in sequence,
|
|
finally, after the SST file is generated, we use `SstFileReader` to check the file's checksum.
|
|
|
|
### Define input
|
|
|
|
We define the database operations in protobuf, each operation has a type of operation and a key value pair,
|
|
see [proto/db_operation.proto](proto/db_operation.proto) for details.
|
|
|
|
### Define tests with the input
|
|
|
|
In [sst_file_writer_fuzzer.cc](sst_file_writer_fuzzer.cc),
|
|
we define the tests to be run on the generated input:
|
|
|
|
```
|
|
DEFINE_PROTO_FUZZER(DBOperations& input) {
|
|
// apply the operations to SstFileWriter and use SstFileReader to verify checksum.
|
|
// ...
|
|
}
|
|
```
|
|
|
|
`SstFileWriter` requires the keys of the operations to be unique and be in ascending order,
|
|
but the fuzzing engine generates the input randomly, so we need to process the generated input before
|
|
passing it to `DEFINE_PROTO_FUZZER`, this is accomplished by registering a post processor:
|
|
|
|
```
|
|
protobuf_mutator::libfuzzer::PostProcessorRegistration<DBOperations>
|
|
```
|
|
|
|
### Compile and link the fuzzer
|
|
|
|
In the rocksdb root directory, compile rocksdb library by `make static_lib`.
|
|
|
|
Go to the `fuzz` directory,
|
|
run `make sst_file_writer_fuzzer` to generate the fuzzer,
|
|
it will compile rocksdb static library, generate protobuf, then compile and link `sst_file_writer_fuzzer`.
|
|
|
|
### Introduce a bug
|
|
|
|
Manually introduce a bug to `SstFileWriter::Put`:
|
|
|
|
```
|
|
diff --git a/table/sst_file_writer.cc b/table/sst_file_writer.cc
|
|
index ab1ee7c4e..c7da9ffa0 100644
|
|
--- a/table/sst_file_writer.cc
|
|
+++ b/table/sst_file_writer.cc
|
|
@@ -277,6 +277,11 @@ Status SstFileWriter::Add(const Slice& user_key, const Slice& value) {
|
|
}
|
|
|
|
Status SstFileWriter::Put(const Slice& user_key, const Slice& value) {
|
|
+ if (user_key.starts_with("!")) {
|
|
+ if (value.ends_with("!")) {
|
|
+ return Status::Corruption("bomb");
|
|
+ }
|
|
+ }
|
|
return rep_->Add(user_key, value, ValueType::kTypeValue);
|
|
}
|
|
```
|
|
|
|
The bug is that for `Put`, if `user_key` starts with `!` and `value` ends with `!`, then corrupt.
|
|
|
|
### Run fuzz testing to catch the bug
|
|
|
|
Run the fuzzer by `time ./sst_file_writer_fuzzer`.
|
|
|
|
Here is the output on my machine:
|
|
|
|
```
|
|
Corruption: bomb
|
|
==59680== ERROR: libFuzzer: deadly signal
|
|
#0 0x109487315 in __sanitizer_print_stack_trace+0x35 (libclang_rt.asan_osx_dynamic.dylib:x86_64+0x4d315)
|
|
#1 0x108d63f18 in fuzzer::PrintStackTrace() FuzzerUtil.cpp:205
|
|
#2 0x108d47613 in fuzzer::Fuzzer::CrashCallback() FuzzerLoop.cpp:232
|
|
#3 0x7fff6af535fc in _sigtramp+0x1c (libsystem_platform.dylib:x86_64+0x35fc)
|
|
#4 0x7ffee720f3ef (<unknown module>)
|
|
#5 0x7fff6ae29807 in abort+0x77 (libsystem_c.dylib:x86_64+0x7f807)
|
|
#6 0x108cf1c4c in TestOneProtoInput(DBOperations&)+0x113c (sst_file_writer_fuzzer:x86_64+0x100302c4c)
|
|
#7 0x108cf09be in LLVMFuzzerTestOneInput+0x16e (sst_file_writer_fuzzer:x86_64+0x1003019be)
|
|
#8 0x108d48ce0 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) FuzzerLoop.cpp:556
|
|
#9 0x108d48425 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool*) FuzzerLoop.cpp:470
|
|
#10 0x108d4a626 in fuzzer::Fuzzer::MutateAndTestOne() FuzzerLoop.cpp:698
|
|
#11 0x108d4b325 in fuzzer::Fuzzer::Loop(std::__1::vector<fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile> >&) FuzzerLoop.cpp:830
|
|
#12 0x108d37fcd in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) FuzzerDriver.cpp:829
|
|
#13 0x108d652b2 in main FuzzerMain.cpp:19
|
|
#14 0x7fff6ad5acc8 in start+0x0 (libdyld.dylib:x86_64+0x1acc8)
|
|
|
|
NOTE: libFuzzer has rudimentary signal handlers.
|
|
Combine libFuzzer with AddressSanitizer or similar for better crash reports.
|
|
SUMMARY: libFuzzer: deadly signal
|
|
MS: 7 Custom-CustomCrossOver-InsertByte-Custom-ChangeBit-Custom-CustomCrossOver-; base unit: 90863b4d83c3f994bba0a417d0c2ee3b68f9e795
|
|
0x6f,0x70,0x65,0x72,0x61,0x74,0x69,0x6f,0x6e,0x73,0x20,0x7b,0xa,0x20,0x20,0x6b,0x65,0x79,0x3a,0x20,0x22,0x21,0x22,0xa,0x20,0x20,0x76,0x61,0x6c,0x75,0x65,0x3a,0x20,0x22,0x21,0x22,0xa,0x20,0x20,0x74,0x79,0x70,0x65,0x3a,0x20,0x50,0x55,0x54,0xa,0x7d,0xa,0x6f,0x70,0x65,0x72,0x61,0x74,0x69,0x6f,0x6e,0x73,0x20,0x7b,0xa,0x20,0x20,0x6b,0x65,0x79,0x3a,0x20,0x22,0x2b,0x22,0xa,0x20,0x20,0x74,0x79,0x70,0x65,0x3a,0x20,0x50,0x55,0x54,0xa,0x7d,0xa,0x6f,0x70,0x65,0x72,0x61,0x74,0x69,0x6f,0x6e,0x73,0x20,0x7b,0xa,0x20,0x20,0x6b,0x65,0x79,0x3a,0x20,0x22,0x2e,0x22,0xa,0x20,0x20,0x74,0x79,0x70,0x65,0x3a,0x20,0x50,0x55,0x54,0xa,0x7d,0xa,0x6f,0x70,0x65,0x72,0x61,0x74,0x69,0x6f,0x6e,0x73,0x20,0x7b,0xa,0x20,0x20,0x6b,0x65,0x79,0x3a,0x20,0x22,0x5c,0x32,0x35,0x33,0x22,0xa,0x20,0x20,0x74,0x79,0x70,0x65,0x3a,0x20,0x50,0x55,0x54,0xa,0x7d,0xa,
|
|
operations {\x0a key: \"!\"\x0a value: \"!\"\x0a type: PUT\x0a}\x0aoperations {\x0a key: \"+\"\x0a type: PUT\x0a}\x0aoperations {\x0a key: \".\"\x0a type: PUT\x0a}\x0aoperations {\x0a key: \"\\253\"\x0a type: PUT\x0a}\x0a
|
|
artifact_prefix='./'; Test unit written to ./crash-a1460be302d09b548e61787178d9edaa40aea467
|
|
Base64: b3BlcmF0aW9ucyB7CiAga2V5OiAiISIKICB2YWx1ZTogIiEiCiAgdHlwZTogUFVUCn0Kb3BlcmF0aW9ucyB7CiAga2V5OiAiKyIKICB0eXBlOiBQVVQKfQpvcGVyYXRpb25zIHsKICBrZXk6ICIuIgogIHR5cGU6IFBVVAp9Cm9wZXJhdGlvbnMgewogIGtleTogIlwyNTMiCiAgdHlwZTogUFVUCn0K
|
|
./sst_file_writer_fuzzer 5.97s user 4.40s system 64% cpu 16.195 total
|
|
```
|
|
|
|
Within 6 seconds, it catches the bug.
|
|
|
|
The input that triggers the bug is persisted in `./crash-a1460be302d09b548e61787178d9edaa40aea467`:
|
|
|
|
```
|
|
$ cat ./crash-a1460be302d09b548e61787178d9edaa40aea467
|
|
operations {
|
|
key: "!"
|
|
value: "!"
|
|
type: PUT
|
|
}
|
|
operations {
|
|
key: "+"
|
|
type: PUT
|
|
}
|
|
operations {
|
|
key: "."
|
|
type: PUT
|
|
}
|
|
operations {
|
|
key: "\253"
|
|
type: PUT
|
|
}
|
|
```
|
|
|
|
### Reproduce the crash to debug
|
|
|
|
The above crash can be reproduced by `./sst_file_writer_fuzzer ./crash-a1460be302d09b548e61787178d9edaa40aea467`,
|
|
so you can debug the crash.
|
|
|
|
## Future Work
|
|
|
|
According to [OSS-Fuzz](https://github.com/google/oss-fuzz),
|
|
`as of June 2020, OSS-Fuzz has found over 20,000 bugs in 300 open source projects.`
|
|
|
|
RocksDB can join OSS-Fuzz together with other open source projects such as sqlite.
|
|
|