rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

History

Peter Dillinger 25a0d0ca30 Fix block checksum for >=4GB, refactor (#6978 ) Summary: Although RocksDB falls over in various other ways with KVs around 4GB or more, this change fixes how XXH32 and XXH64 were being called by the block checksum code to support >= 4GB in case that should ever happen, or the code copied for other uses. This change is not a schema compatibility issue because the checksum verification code would checksum the first (block_size + 1) mod 2^32 bytes while the checksum construction code would checksum the first block_size mod 2^32 plus the compression type byte, meaning the XXH32/64 checksums for >=4GB block would not match about 255/256 times. While touching this code, I refactored to consolidate redundant implementations, improving diagnostics and performance tracking in some cases. Also used less confusing language in those diagnostics. Makes https://github.com/facebook/rocksdb/issues/6875 obsolete. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6978 Test Plan: I was able to write a test for this using an SST file writer and VerifyChecksum in a reader. The test fails before the fix, though I'm leaving the test disabled because I don't think it's worth the expense of running regularly. Reviewed By: gg814 Differential Revision: D22143260 Pulled By: pdillinger fbshipit-source-id: 982993d16134e8c50bea2269047f901c1783726e		5 years ago
..
adaptive	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
block_based	Fix block checksum for >=4GB, refactor (#6978 )	5 years ago
cuckoo	Store DB identity and DB session ID in SST files (#6983 )	5 years ago
plain	Store DB identity and DB session ID in SST files (#6983 )	5 years ago
block_fetcher.cc	Fix block checksum for >=4GB, refactor (#6978 )	5 years ago
block_fetcher.h	Reduce memory copies when fetching and uncompressing blocks from SST files (#6689 )	5 years ago
block_fetcher_test.cc	skip direct I/O tests in rocksdb lite (#6867 )	5 years ago
cleanable_test.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
format.cc	Add tests for compression failure in BlockBasedTableBuilder (#6709 )	5 years ago
format.h	Fix block checksum for >=4GB, refactor (#6978 )	5 years ago
get_context.cc	Add timestamp to delete (#6253 )	5 years ago
get_context.h	Stats for redundant insertions into block cache (#6681 )	5 years ago
internal_iterator.h	Properly report IO errors when IndexType::kBinarySearchWithFirstKey is used (#6621 )	5 years ago
iter_heap.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
iterator.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
iterator_wrapper.h	Properly report IO errors when IndexType::kBinarySearchWithFirstKey is used (#6621 )	5 years ago
merger_test.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
merging_iterator.cc	Properly report IO errors when IndexType::kBinarySearchWithFirstKey is used (#6621 )	5 years ago
merging_iterator.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
meta_blocks.cc	Store DB identity and DB session ID in SST files (#6983 )	5 years ago
meta_blocks.h	sst_dump to reduce number of file reads (#6836 )	5 years ago
mock_table.cc	Pass a timeout to FileSystem for random reads (#6751 )	5 years ago
mock_table.h	Remove unnecessary inclusion of version_edit.h in env (#6952 )	5 years ago
multiget_context.h	Allow MultiGet users to limit cumulative value size (#6826 )	5 years ago
persistent_cache_helper.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
persistent_cache_helper.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
persistent_cache_options.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
scoped_arena_iterator.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
sst_file_reader.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
sst_file_reader_test.cc	Allow table/sst_file_reader_test.cc to use custom Env (#6536 )	5 years ago
sst_file_writer.cc	Store DB identity and DB session ID in SST files (#6983 )	5 years ago
sst_file_writer_collectors.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
table_builder.h	Store DB identity and DB session ID in SST files (#6983 )	5 years ago
table_properties.cc	Store DB identity and DB session ID in SST files (#6983 )	5 years ago
table_properties_internal.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
table_reader.h	For ApproximateSizes, pro-rate table metadata size over data blocks (#6784 )	5 years ago
table_reader_bench.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
table_reader_caller.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
table_test.cc	Fix uninitialized memory read in table_test (#6980 )	5 years ago
two_level_iterator.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
two_level_iterator.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago