Fuzzing RocksDB (#7685)

Summary:
This is the initial PR to support adding fuzz tests to RocksDB.
It includes the necessary build infrastructure, and includes an example fuzzer.
There is also a README serving as the tutorial for how to add more tests.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7685

Test Plan: Manually build and run the fuzz test according to README.

Reviewed By: pdillinger

Differential Revision: D25013847

Pulled By: cheng-chang

fbshipit-source-id: c91e3b337398d7f4d8f769fd5091cd080487b171
main
Cheng Chang 4 years ago committed by Facebook GitHub Bot
parent 84a700819e
commit 699411b2ca
  1. 3
      .gitignore
  2. 38
      fuzz/Makefile
  3. 158
      fuzz/README.md
  4. 28
      fuzz/proto/db_operation.proto
  5. 101
      fuzz/sst_file_writer_fuzzer.cc

3
.gitignore vendored

@ -90,3 +90,6 @@ buckifier/__pycache__
compile_commands.json
clang-format-diff.py
.py3/
fuzz/proto/gen/
fuzz/crash-*

@ -0,0 +1,38 @@
# Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
# This source code is licensed under both the GPLv2 (found in the
# COPYING file in the root directory) and Apache 2.0 License
# (found in the LICENSE.Apache file in the root directory).
CWD = $(shell pwd)
PROTOBUF_CFLAGS = `pkg-config --cflags protobuf`
PROTOBUF_LDFLAGS = `pkg-config --libs protobuf`
PROTOBUF_MUTATOR_CFLAGS = `pkg-config --cflags libprotobuf-mutator`
PROTOBUF_MUTATOR_LDFLAGS = `pkg-config --libs libprotobuf-mutator`
ROCKSDB_INCLUDE_DIR = $(CWD)/../include
ROCKSDB_LIB_DIR = $(CWD)/..
PROTO_IN = $(CWD)/proto
PROTO_OUT = $(CWD)/proto/gen
CC = clang++
CCFLAGS += -std=c++14 -Wall -fsanitize=address,fuzzer
CFLAGS += $(PROTOBUF_CFLAGS) $(PROTOBUF_MUTATOR_CFLAGS) -I$(PROTO_OUT) -I$(ROCKSDB_INCLUDE_DIR)
LDFLAGS += $(PROTOBUF_LDFLAGS) $(PROTOBUF_MUTATOR_LDFLAGS) -L$(ROCKSDB_LIB_DIR) -lrocksdb -lz -lbz2
.PHONY: librocksdb gen_proto
librocksdb:
cd $(CWD)/.. && make static_lib
gen_proto:
mkdir -p $(PROTO_OUT)
protoc \
--proto_path=$(PROTO_IN) \
--cpp_out=$(PROTO_OUT) \
$(PROTO_IN)/*.proto
sst_file_writer_fuzzer: librocksdb gen_proto sst_file_writer_fuzzer.cc proto/gen/db_operation.pb.cc
$(CC) $(CCFLAGS) $(CFLAGS) $(LDFLAGS) -o sst_file_writer_fuzzer sst_file_writer_fuzzer.cc proto/gen/db_operation.pb.cc

@ -0,0 +1,158 @@
# Fuzzing RocksDB
## Overview
This directory contains [fuzz tests](https://en.wikipedia.org/wiki/Fuzzing) for RocksDB.
RocksDB testing infrastructure currently includes unit tests and [stress tests](https://github.com/facebook/rocksdb/wiki/Stress-test),
we hope fuzz testing can catch more bugs.
## Prerequisite
We use [LLVM libFuzzer](http://llvm.org/docs/LibFuzzer.html) as the fuzzying engine,
so make sure you have [clang](https://clang.llvm.org/get_started.html) as your compiler.
Some tests rely on [structure aware fuzzing](https://github.com/google/fuzzing/blob/master/docs/structure-aware-fuzzing.md).
We use [protobuf](https://developers.google.com/protocol-buffers) to define structured input to the fuzzer,
and use [libprotobuf-mutator](https://github.com/google/libprotobuf-mutator) as the custom libFuzzer mutator.
So make sure you have protobuf and libprotobuf-mutator installed, and make sure `pkg-config` can find them.
## Example
This example shows you how to do structure aware fuzzing to `rocksdb::SstFileWriter`.
After walking through the steps to create the fuzzer, we'll introduce a bug into `rocksdb::SstFileWriter::Put`,
then show that the fuzzer can catch the bug.
### Design the test
We want the fuzzing engine to automatically generate a list of database operations,
then we apply these operations to `SstFileWriter` in sequence,
finally, after the SST file is generated, we use `SstFileReader` to check the file's checksum.
### Define input
We define the database operations in protobuf, each operation has a type of operation and a key value pair,
see [proto/db_operation.proto](proto/db_operation.proto) for details.
### Define tests with the input
In [sst_file_writer_fuzzer.cc](sst_file_writer_fuzzer.cc),
we define the tests to be run on the generated input:
```
DEFINE_PROTO_FUZZER(DBOperations& input) {
// apply the operations to SstFileWriter and use SstFileReader to verify checksum.
// ...
}
```
`SstFileWriter` requires the keys of the operations to be unique and be in ascending order,
but the fuzzing engine generates the input randomly, so we need to process the generated input before
passing it to `DEFINE_PROTO_FUZZER`, this is accomplished by registering a post processor:
```
protobuf_mutator::libfuzzer::PostProcessorRegistration<DBOperations>
```
### Compile and link the fuzzer
Go to the `fuzz` directory,
run `make sst_file_writer_fuzzer` to generate the fuzzer,
it will compile rocksdb static library, generate protobuf, then compile and link `sst_file_writer_fuzzer`.
### Introduce a bug
Manually introduce a bug to `SstFileWriter::Put`:
```
diff --git a/table/sst_file_writer.cc b/table/sst_file_writer.cc
index ab1ee7c4e..c7da9ffa0 100644
--- a/table/sst_file_writer.cc
+++ b/table/sst_file_writer.cc
@@ -277,6 +277,11 @@ Status SstFileWriter::Add(const Slice& user_key, const Slice& value) {
}
Status SstFileWriter::Put(const Slice& user_key, const Slice& value) {
+ if (user_key.starts_with("!")) {
+ if (value.ends_with("!")) {
+ return Status::Corruption("bomb");
+ }
+ }
return rep_->Add(user_key, value, ValueType::kTypeValue);
}
```
The bug is that for `Put`, if `user_key` starts with `!` and `value` ends with `!`, then corrupt.
### Run fuzz testing to catch the bug
Run the fuzzer by `time ./sst_file_writer_fuzzer`.
Here is the output on my machine:
```
Corruption: bomb
==59680== ERROR: libFuzzer: deadly signal
#0 0x109487315 in __sanitizer_print_stack_trace+0x35 (libclang_rt.asan_osx_dynamic.dylib:x86_64+0x4d315)
#1 0x108d63f18 in fuzzer::PrintStackTrace() FuzzerUtil.cpp:205
#2 0x108d47613 in fuzzer::Fuzzer::CrashCallback() FuzzerLoop.cpp:232
#3 0x7fff6af535fc in _sigtramp+0x1c (libsystem_platform.dylib:x86_64+0x35fc)
#4 0x7ffee720f3ef (<unknown module>)
#5 0x7fff6ae29807 in abort+0x77 (libsystem_c.dylib:x86_64+0x7f807)
#6 0x108cf1c4c in TestOneProtoInput(DBOperations&)+0x113c (sst_file_writer_fuzzer:x86_64+0x100302c4c)
#7 0x108cf09be in LLVMFuzzerTestOneInput+0x16e (sst_file_writer_fuzzer:x86_64+0x1003019be)
#8 0x108d48ce0 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) FuzzerLoop.cpp:556
#9 0x108d48425 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool*) FuzzerLoop.cpp:470
#10 0x108d4a626 in fuzzer::Fuzzer::MutateAndTestOne() FuzzerLoop.cpp:698
#11 0x108d4b325 in fuzzer::Fuzzer::Loop(std::__1::vector<fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile> >&) FuzzerLoop.cpp:830
#12 0x108d37fcd in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) FuzzerDriver.cpp:829
#13 0x108d652b2 in main FuzzerMain.cpp:19
#14 0x7fff6ad5acc8 in start+0x0 (libdyld.dylib:x86_64+0x1acc8)
NOTE: libFuzzer has rudimentary signal handlers.
Combine libFuzzer with AddressSanitizer or similar for better crash reports.
SUMMARY: libFuzzer: deadly signal
MS: 7 Custom-CustomCrossOver-InsertByte-Custom-ChangeBit-Custom-CustomCrossOver-; base unit: 90863b4d83c3f994bba0a417d0c2ee3b68f9e795
0x6f,0x70,0x65,0x72,0x61,0x74,0x69,0x6f,0x6e,0x73,0x20,0x7b,0xa,0x20,0x20,0x6b,0x65,0x79,0x3a,0x20,0x22,0x21,0x22,0xa,0x20,0x20,0x76,0x61,0x6c,0x75,0x65,0x3a,0x20,0x22,0x21,0x22,0xa,0x20,0x20,0x74,0x79,0x70,0x65,0x3a,0x20,0x50,0x55,0x54,0xa,0x7d,0xa,0x6f,0x70,0x65,0x72,0x61,0x74,0x69,0x6f,0x6e,0x73,0x20,0x7b,0xa,0x20,0x20,0x6b,0x65,0x79,0x3a,0x20,0x22,0x2b,0x22,0xa,0x20,0x20,0x74,0x79,0x70,0x65,0x3a,0x20,0x50,0x55,0x54,0xa,0x7d,0xa,0x6f,0x70,0x65,0x72,0x61,0x74,0x69,0x6f,0x6e,0x73,0x20,0x7b,0xa,0x20,0x20,0x6b,0x65,0x79,0x3a,0x20,0x22,0x2e,0x22,0xa,0x20,0x20,0x74,0x79,0x70,0x65,0x3a,0x20,0x50,0x55,0x54,0xa,0x7d,0xa,0x6f,0x70,0x65,0x72,0x61,0x74,0x69,0x6f,0x6e,0x73,0x20,0x7b,0xa,0x20,0x20,0x6b,0x65,0x79,0x3a,0x20,0x22,0x5c,0x32,0x35,0x33,0x22,0xa,0x20,0x20,0x74,0x79,0x70,0x65,0x3a,0x20,0x50,0x55,0x54,0xa,0x7d,0xa,
operations {\x0a key: \"!\"\x0a value: \"!\"\x0a type: PUT\x0a}\x0aoperations {\x0a key: \"+\"\x0a type: PUT\x0a}\x0aoperations {\x0a key: \".\"\x0a type: PUT\x0a}\x0aoperations {\x0a key: \"\\253\"\x0a type: PUT\x0a}\x0a
artifact_prefix='./'; Test unit written to ./crash-a1460be302d09b548e61787178d9edaa40aea467
Base64: b3BlcmF0aW9ucyB7CiAga2V5OiAiISIKICB2YWx1ZTogIiEiCiAgdHlwZTogUFVUCn0Kb3BlcmF0aW9ucyB7CiAga2V5OiAiKyIKICB0eXBlOiBQVVQKfQpvcGVyYXRpb25zIHsKICBrZXk6ICIuIgogIHR5cGU6IFBVVAp9Cm9wZXJhdGlvbnMgewogIGtleTogIlwyNTMiCiAgdHlwZTogUFVUCn0K
./sst_file_writer_fuzzer 5.97s user 4.40s system 64% cpu 16.195 total
```
Within 6 seconds, it catches the bug.
The input that triggers the bug is persisted in `./crash-a1460be302d09b548e61787178d9edaa40aea467`:
```
$ cat ./crash-a1460be302d09b548e61787178d9edaa40aea467
operations {
key: "!"
value: "!"
type: PUT
}
operations {
key: "+"
type: PUT
}
operations {
key: "."
type: PUT
}
operations {
key: "\253"
type: PUT
}
```
### Reproduce the crash to debug
The above crash can be reproduced by `./sst_file_writer_fuzzer ./crash-a1460be302d09b548e61787178d9edaa40aea467`,
so you can debug the crash.
## Future Work
According to [OSS-Fuzz](https://github.com/google/oss-fuzz),
`as of June 2020, OSS-Fuzz has found over 20,000 bugs in 300 open source projects.`
RocksDB can join OSS-Fuzz together with other open source projects such as sqlite.

@ -0,0 +1,28 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
// Defines database operations.
// Each operation is a key-value pair and an operation type.
syntax = "proto2";
enum OpType {
PUT = 0;
MERGE = 1;
DELETE = 2;
DELETE_RANGE = 3;
}
message DBOperation {
required string key = 1;
// value is ignored for DELETE.
// [key, value] is the range for DELETE_RANGE.
optional string value = 2;
required OpType type = 3;
}
message DBOperations {
repeated DBOperation operations = 1;
}

@ -0,0 +1,101 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
#include <algorithm>
#include <iostream>
#include "proto/gen/db_operation.pb.h"
#include "rocksdb/file_system.h"
#include "rocksdb/sst_file_reader.h"
#include "rocksdb/sst_file_writer.h"
#include "src/libfuzzer/libfuzzer_macro.h"
// Keys in SST file writer operations must be unique and in ascending order.
// For each DBOperation generated by the fuzzer, this function is called on
// it to deduplicate and sort the keys in the DBOperations.
protobuf_mutator::libfuzzer::PostProcessorRegistration<DBOperations> reg = {
[](DBOperations* input, unsigned int /* seed */) {
const rocksdb::Comparator* comparator = rocksdb::BytewiseComparator();
auto ops = input->mutable_operations();
std::sort(ops->begin(), ops->end(),
[&comparator](const DBOperation& a, const DBOperation& b) {
return comparator->Compare(a.key(), b.key()) < 0;
});
auto last = std::unique(
ops->begin(), ops->end(),
[&comparator](const DBOperation& a, const DBOperation& b) {
return comparator->Compare(a.key(), b.key()) == 0;
});
ops->erase(last, ops->end());
}};
#define CHECK_OK(status) \
if (!status.ok()) { \
std::cerr << status.ToString() << std::endl; \
abort(); \
}
// Fuzzes DB operations as input, let SstFileWriter generate a SST file
// according to the operations, then let SstFileReader read the generated SST
// file to check its checksum.
DEFINE_PROTO_FUZZER(DBOperations& input) {
if (input.operations().empty()) {
return;
}
std::string sstfile;
{
auto fs = rocksdb::FileSystem::Default();
std::string dir;
rocksdb::IOOptions opt;
rocksdb::IOStatus s = fs->GetTestDirectory(opt, &dir, nullptr);
CHECK_OK(s);
sstfile = dir + "/SstFileWriterFuzzer.sst";
}
// Generate sst file.
rocksdb::Options options;
rocksdb::EnvOptions env_options;
rocksdb::SstFileWriter writer(env_options, options);
rocksdb::Status s = writer.Open(sstfile);
CHECK_OK(s);
for (const DBOperation& op : input.operations()) {
switch (op.type()) {
case OpType::PUT:
s = writer.Put(op.key(), op.value());
CHECK_OK(s);
break;
case OpType::MERGE:
s = writer.Merge(op.key(), op.value());
CHECK_OK(s);
break;
case OpType::DELETE:
s = writer.Delete(op.key());
CHECK_OK(s);
break;
case OpType::DELETE_RANGE:
s = writer.DeleteRange(op.key(), op.value());
CHECK_OK(s);
break;
default:
std::cerr << "Unsupported operation" << static_cast<int>(op.type());
return;
}
}
rocksdb::ExternalSstFileInfo info;
s = writer.Finish(&info);
CHECK_OK(s);
// Verify checksum.
rocksdb::SstFileReader reader(options);
s = reader.Open(sstfile);
CHECK_OK(s);
s = reader.VerifyChecksum();
CHECK_OK(s);
// Delete sst file.
remove(sstfile.c_str());
}
Loading…
Cancel
Save