rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

Zhichao Cao b3585a11b4 Ingest SST files with checksum information (#6891 ) Summary: Application can ingest SST files with file checksum information, such that during ingestion, DB is able to check data integrity and identify of the SST file. The PR introduces generate_and_verify_file_checksum to IngestExternalFileOption to control if the ingested checksum information should be verified with the generated checksum. 1. If generate_and_verify_file_checksum options is FALSE: 1) if DB does not enable SST file checksum, the checksum information ingested will be ignored; 2) if DB enables the SST file checksum and the checksum function name matches the checksum function name in DB, we trust the ingested checksum, store it in Manifest. If the checksum function name does not match, we treat that as an error and fail the IngestExternalFile() call. 2. If generate_and_verify_file_checksum options is TRUE: 1) if DB does not enable SST file checksum, the checksum information ingested will be ignored; 2) if DB enable the SST file checksum, we will use the checksum generator from DB to calculate the checksum for each ingested SST files after they are copied or moved. Then, compare the checksum results with the ingested checksum information: _A)_ if the checksum function name does not match, _verification always report true_ and we store the DB generated checksum information in Manifest. _B)_ if the checksum function name mach, and checksum match, ingestion continues and stores the checksum information in the Manifest. Otherwise, terminate file ingestion and report file corruption. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6891 Test Plan: added unit test, pass make asan_check Reviewed By: pdillinger Differential Revision: D21935988 Pulled By: zhichao-cao fbshipit-source-id: 7b55f486632db467e76d72602218d0658aa7f6ed		4 years ago
.circleci	Introduce some Linux build to CircleCI (#6937 )	4 years ago
.github/workflows	Clean up some code related to file checksums (#6861 )	5 years ago
buckifier	Directly use unit test tempalte buck (#6926 )	5 years ago
build_tools	Allow missing "unversioned" python, as in CentOS 8 (#6883 )	5 years ago
cache	Revert "Update googletest from 1.8.1 to 1.10.0 (#6808 )" (#6923 )	5 years ago
cmake	Add `find_dependency()` in cmake config file. (#6791 )	5 years ago
coverage	Find the correct gcov (#6904 )	5 years ago
db	Ingest SST files with checksum information (#6891 )	4 years ago
db_stress_tool	Fix potential overflow of unsigned type in for loop (#6902 )	5 years ago
docs	Log warning for high bits/key in legacy Bloom filter (#6312 )	5 years ago
env	Close file to avoid file-descriptor leakage (#6936 )	5 years ago
examples	Add missing my_pid to fprintf in multi_process_example (#6731 )	5 years ago
file	Ingest SST files with checksum information (#6891 )	4 years ago
hdfs	prototype status check enforcement (#6798 )	5 years ago
include/rocksdb	Ingest SST files with checksum information (#6891 )	4 years ago
java	Add logs and stats in DeleteScheduler (#6927 )	5 years ago
logging	Fix info log source file display length (#5824 )	5 years ago
memory	C++20 compatibility (#6697 )	5 years ago
memtable	Fix more defects reported by Coverity Scan (#6935 )	5 years ago
monitoring	Add logs and stats in DeleteScheduler (#6927 )	5 years ago
options	make L0 index/filter pinned memory usage predictable (#6911 )	4 years ago
port	Fix more defects reported by Coverity Scan (#6935 )	5 years ago
table	save a key comparison in block seeks (#6646 )	4 years ago
test_util	Check iterator status BlockBasedTableReader::VerifyChecksumInBlocks() (#6909 )	5 years ago
third-party	Revert "Update googletest from 1.8.1 to 1.10.0 (#6808 )" (#6923 )	5 years ago
tools	Add --version and --help to ldb and sst_dump (#6951 )	4 years ago
trace_replay	Fix more defects reported by Coverity Scan (#6935 )	5 years ago
util	Fix ThreadLocalTest.SequentialReadWriteTest failure when running individually (#6929 )	5 years ago
utilities	Move blob_log_{format,reader,writer}.{cc,h} to db/blob/ (#6960 )	4 years ago
.clang-format	A script that automatically reformat affected lines	11 years ago
.gitignore	Allow missing "unversioned" python, as in CentOS 8 (#6883 )	5 years ago
.lgtm.yml	Create lgtm.yml for LGTM.com C/C++ analysis (#4058 )	6 years ago
.travis.yml	Make sure core components not depend on gtest (#6921 )	5 years ago
.watchmanconfig	Added .watchmanconfig file to rocksdb repo (#5593 )	5 years ago
AUTHORS	Update RocksDB Authors File	7 years ago
CMakeLists.txt	Move blob_log_{format,reader,writer}.{cc,h} to db/blob/ (#6960 )	4 years ago
CODE_OF_CONDUCT.md	Adopt Contributor Covenant	5 years ago
CONTRIBUTING.md	Add Code of Conduct	7 years ago
COPYING	Add GPLv2 as an alternative license.	8 years ago
DEFAULT_OPTIONS_HISTORY.md	options.delayed_write_rate use the rate of rate_limiter by default.	8 years ago
DUMP_FORMAT.md	First version of rocksdb_dump and rocksdb_undump.	10 years ago
HISTORY.md	Ingest SST files with checksum information (#6891 )	4 years ago
INSTALL.md	Update the version of the dependencies used by the RocksJava static build (#4761 )	6 years ago
LANGUAGE-BINDINGS.md	LANGUAGE-BINDINGS.md: mention python-rocksdb	6 years ago
LICENSE.Apache	Change RocksDB License	7 years ago
LICENSE.leveldb	Add back the LevelDB license file	7 years ago
Makefile	Introduce some Linux build to CircleCI (#6937 )	4 years ago
README.md	Add Slack forum to README (#6773 )	5 years ago
ROCKSDB_LITE.md	Fix some typos in comments and docs.	7 years ago
TARGETS	Move blob_log_{format,reader,writer}.{cc,h} to db/blob/ (#6960 )	4 years ago
USERS.md	Add YugabyteDB to USERS (#6786 )	5 years ago
Vagrantfile	Adding CentOS 7 Vagrantfile & build script	7 years ago
WINDOWS_PORT.md	#5145 , rename port/dirent.h to port/port_dirent.h to avoid compile err when use port dir as header dir output (#5152 )	6 years ago
appveyor.yml	C++20 compatibility (#6697 )	5 years ago
defs.bzl	Make testpilot recognize that these tests have coverage instrumentation	5 years ago
issue_template.md	Add Google Group to Issue Template	5 years ago
src.mk	Move blob_log_{format,reader,writer}.{cc,h} to db/blob/ (#6960 )	4 years ago
thirdparty.inc	Fix build jemalloc api (#5470 )	5 years ago

README.md

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/master/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.

Design discussions are conducted in https://www.facebook.com/groups/rocksdb.dev/ and https://rocksdb.slack.com/

License

RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.