fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Andrew Kryczka 78ee8564ad Integrity protection for live updates to WriteBatch (#7748) ... Summary: This PR adds the foundation classes for key-value integrity protection and the first use case: protecting live updates from the source buffers added to `WriteBatch` through the destination buffer in `MemTable`. The width of the protection info is not yet configurable -- only eight bytes per key is supported. This PR allows users to enable protection by constructing `WriteBatch` with `protection_bytes_per_key == 8`. It does not yet expose a way for users to get integrity protection via other write APIs (e.g., `Put()`, `Merge()`, `Delete()`, etc.). The foundation classes (`ProtectionInfo.*`) embed the coverage info in their type, and provide `Protect.*()` and `Strip.*()` functions to navigate between types with different coverage. For making bytes per key configurable (for powers of two up to eight) in the future, these classes are templated on the unsigned integer type used to store the protection info. That integer contains the XOR'd result of hashes with independent seeds for all covered fields. For integer fields, the hash is computed on the raw unadjusted bytes, so the result is endian-dependent. The most significant bytes are truncated when the hash value (8 bytes) is wider than the protection integer. When `WriteBatch` is constructed with `protection_bytes_per_key == 8`, we hold a `ProtectionInfoKVOTC` (i.e., one that covers key, value, optype aka `ValueType`, timestamp, and CF ID) for each entry added to the batch. The protection info is generated from the original buffers passed by the user, as well as the original metadata generated internally. When writing to memtable, each entry is transformed to a `ProtectionInfoKVOTS` (i.e., dropping coverage of CF ID and adding coverage of sequence number), since at that point we know the sequence number, and have already selected a memtable corresponding to a particular CF. This protection info is verified once the entry is encoded in the `MemTable` buffer. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7748 Test Plan: - an integration test to verify a wide variety of single-byte changes to the encoded `MemTable` buffer are caught - add to stress/crash test to verify it works in variety of configs/operations without intentional corruption - [deferred] unit tests for `ProtectionInfo.*` classes for edge cases like KV swap, `SliceParts` and `Slice` APIs are interchangeable, etc. Reviewed By: pdillinger Differential Revision: D25754492 Pulled By: ajkr fbshipit-source-id: e481bac6c03c2ab268be41359730f1ceb9964866		3 years ago
..
blob_constants.h	Move BlobDB related files under db/ to db/blob/ (#6519)	4 years ago
blob_file_addition.cc	Move BlobDB related files under db/ to db/blob/ (#6519)	4 years ago
blob_file_addition.h	Move BlobDB related files under db/ to db/blob/ (#6519)	4 years ago
blob_file_addition_test.cc	Move BlobDB related files under db/ to db/blob/ (#6519)	4 years ago
blob_file_builder.cc	Add a SystemClock class to capture the time functions of an Env (#7858)	3 years ago
blob_file_builder.h	Add a SystemClock class to capture the time functions of an Env (#7858)	3 years ago
blob_file_builder_test.cc	Add a SystemClock class to capture the time functions of an Env (#7858)	3 years ago
blob_file_cache.cc	Integrity protection for live updates to WriteBatch (#7748)	3 years ago
blob_file_cache.h	Introduce BlobFileCache and add support for blob files to Get() (#7540)	4 years ago
blob_file_cache_test.cc	Add a SystemClock class to capture the time functions of an Env (#7858)	3 years ago
blob_file_garbage.cc	Move BlobDB related files under db/ to db/blob/ (#6519)	4 years ago
blob_file_garbage.h	Move BlobDB related files under db/ to db/blob/ (#6519)	4 years ago
blob_file_garbage_test.cc	Move BlobDB related files under db/ to db/blob/ (#6519)	4 years ago
blob_file_meta.cc	Maintain the set of linked SSTs in BlobFileMetaData (#6945)	4 years ago
blob_file_meta.h	Maintain the set of linked SSTs in BlobFileMetaData (#6945)	4 years ago
blob_file_reader.cc	Add a SystemClock class to capture the time functions of an Env (#7858)	3 years ago
blob_file_reader.h	Introduce a blob file reader class (#7461)	4 years ago
blob_file_reader_test.cc	Add a SystemClock class to capture the time functions of an Env (#7858)	3 years ago
blob_index.h	Introduce a blob file reader class (#7461)	4 years ago
blob_log_format.cc	Introduce BlobFileCache and add support for blob files to Get() (#7540)	4 years ago
blob_log_format.h	Introduce BlobFileCache and add support for blob files to Get() (#7540)	4 years ago
blob_log_sequential_reader.cc	Add a SystemClock class to capture the time functions of an Env (#7858)	3 years ago
blob_log_sequential_reader.h	Add a SystemClock class to capture the time functions of an Env (#7858)	3 years ago
blob_log_writer.cc	Add a SystemClock class to capture the time functions of an Env (#7858)	3 years ago
blob_log_writer.h	Add a SystemClock class to capture the time functions of an Env (#7858)	3 years ago
db_blob_basic_test.cc	Add initial blob support to batched MultiGet (#7766)	4 years ago
db_blob_index_test.cc	Add further tests to ASSERT_STATUS_CHECKED (2) (#7698)	4 years ago