rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

History

Yanqin Jin 2309fd63bf Update column families' log number altogether after flushing during recovery (#5856 ) Summary: A bug occasionally shows up in crash test, and https://github.com/facebook/rocksdb/issues/5851 reproduces it. The bug can surface in the following way. 1. Database has multiple column families. 2. Between one DB restart, the last log file is corrupted in the middle (not the tail) 3. During restart, DB crashes between flushing between two column families. Then DB will fail to be opened again with error "SST file is ahead of WALs". Solution is to update the log number associated with each column family altogether after flushing all column families' memtables. The version edits should be written to a new MANIFEST. Only after writing to all these version edits succeed does RocksDB (atomically) points the CURRENT file to the new MANIFEST. Test plan (on devserver): ``` $make all && make check ``` Specifically ``` $make db_test2 $./db_test2 --gtest_filter=DBTest2.CrashInRecoveryMultipleCF ``` Also checked for compatibility as follows. Use this branch, run DBTest2.CrashInRecoveryMultipleCF and preserve the db directory. Then checkout 5.4, build ldb, and dump the MANIFEST. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5856 Differential Revision: D17620818 Pulled By: riversand963 fbshipit-source-id: b52ce5969c9a8052cacec2bd805fcfb373589039		5 years ago
..
compaction	Fix clang analyzer error (#5924 )	5 years ago
db_impl	Update column families' log number altogether after flushing during recovery (#5856 )	5 years ago
arena_wrapped_db_iter.cc	Apply formatter on recent 45 commits. (#5827 )	5 years ago
arena_wrapped_db_iter.h	Refactor ArenaWrappedDBIter into separate files (#5801 )	5 years ago
blob_index.h	Support decoding blob indexes in sst_dump (#5926 )	5 years ago
builder.cc	BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903 )	5 years ago
builder.h	Move some logging related files to logging/ (#5387 )	6 years ago
c.cc	Remove snap_refresh_nanos option (#5826 )	5 years ago
c_test.c	Add C binding for secondary instance (#5505 )	5 years ago
column_family.cc	Apply formatter to recent 200+ commits. (#5830 )	5 years ago
column_family.h	Bump up memory order of ref counting of ColumnFamilyData (#5723 )	5 years ago
column_family_test.cc	Refactor trimming logic for immutable memtables (#5022 )	5 years ago
compact_files_test.cc	Organizing rocksdb/db directory	6 years ago
compacted_db_impl.cc	New API to get all merge operands for a Key (#5604 )	5 years ago
compacted_db_impl.h	Use delete to disable automatic generated methods. (#5009 )	5 years ago
comparator_db_test.cc	Move some memory related files from util/ to memory/ (#5382 )	6 years ago
convenience.cc	Do readahead in VerifyChecksum() (#5713 )	5 years ago
corruption_test.cc	Fix VerifyChecksum readahead with mmap mode (#5945 )	5 years ago
cuckoo_table_db_test.cc	Fix memory leak on error opening PlainTable (#5951 )	5 years ago
db_basic_test.cc	Misc hashing updates / upgrades (#5909 )	5 years ago
db_blob_index_test.cc	Refactor ArenaWrappedDBIter into separate files (#5801 )	5 years ago
db_block_cache_test.cc	Charge block cache for cache internal usage (#5797 )	5 years ago
db_bloom_filter_test.cc	Unordered Writes (#5218 )	6 years ago
db_compaction_filter_test.cc	upgrade gtest 1.7.0 => 1.8.1 for json result writing	5 years ago
db_compaction_test.cc	Apply formatter to recent 200+ commits. (#5830 )	5 years ago
db_dynamic_level_test.cc	Fix flaky DBDynamicLevelTest.DynamicLevelMaxBytesBase2 (#4668 )	6 years ago
db_encryption_test.cc	Fix EncryptedEnv assert (#5735 )	5 years ago
db_filesnapshot.cc	Apply formatter to recent 200+ commits. (#5830 )	5 years ago
db_flush_test.cc	Fix OnFlushCompleted fired before flush result write to MANIFEST (#5908 )	5 years ago
db_info_dumper.cc	Apply formatter to recent 200+ commits. (#5830 )	5 years ago
db_info_dumper.h	Change RocksDB License	7 years ago
db_inplace_update_test.cc	Change RocksDB License	7 years ago
db_io_failure_test.cc	Disable DBIOFailureTest.NoSpaceCompactRange in LITE (#4596 )	6 years ago
db_iter.cc	Improve readability of DBIter's two seek functions (#5794 )	5 years ago
db_iter.h	Apply formatter on recent 45 commits. (#5827 )	5 years ago
db_iter_stress_test.cc	Move some memory related files from util/ to memory/ (#5382 )	6 years ago
db_iter_test.cc	Move some memory related files from util/ to memory/ (#5382 )	6 years ago
db_iterator_test.cc	Fix data block upper bound checking for iterator reseek case (#5883 )	5 years ago
db_log_iter_test.cc	Apply modernize-use-override (2nd iteration)	6 years ago
db_memtable_test.cc	upgrade gtest 1.7.0 => 1.8.1 for json result writing	5 years ago
db_merge_operand_test.cc	New API to get all merge operands for a Key (#5604 )	5 years ago
db_merge_operator_test.cc	New API to get all merge operands for a Key (#5604 )	5 years ago
db_options_test.cc	upgrade gtest 1.7.0 => 1.8.1 for json result writing	5 years ago
db_properties_test.cc	Refactor/consolidate legacy Bloom implementation details (#5784 )	5 years ago
db_range_del_test.cc	Add test showing range tombstones can create excessively large compactions (#5956 )	5 years ago
db_sst_test.cc	Fix bugs in WAL trash file handling (#5520 )	5 years ago
db_statistics_test.cc	Make statistics's stats_level change thread-safe (#5030 )	6 years ago
db_table_properties_test.cc	upgrade gtest 1.7.0 => 1.8.1 for json result writing	5 years ago
db_tailing_iter_test.cc	Remove managed iterator	6 years ago
db_test.cc	Apply formatter to recent 200+ commits. (#5830 )	5 years ago
db_test2.cc	Update column families' log number altogether after flushing during recovery (#5856 )	5 years ago
db_test_util.cc	Apply formatter to recent 200+ commits. (#5830 )	5 years ago
db_test_util.h	Support loading custom objects in unit tests (#5676 )	5 years ago
db_universal_compaction_test.cc	Apply formatter to recent 200+ commits. (#5830 )	5 years ago
db_wal_test.cc	Adding DB::GetCurrentWalFile() API as a repliction/backup helper (#5765 )	5 years ago
db_write_test.cc	Move some memory related files from util/ to memory/ (#5382 )	6 years ago
dbformat.cc	Apply formatter to recent 200+ commits. (#5830 )	5 years ago
dbformat.h	Use delete to disable automatic generated methods. (#5009 )	5 years ago
dbformat_test.cc	Move some logging related files to logging/ (#5387 )	6 years ago
deletefile_test.cc	Refactor deletefile_test.cc (#5822 )	5 years ago
error_handler.cc	Make format	6 years ago
error_handler.h	Fix typos in comments (#4456 )	6 years ago
error_handler_test.cc	Move test related files under util/ to test_util/ (#5377 )	6 years ago
event_helpers.cc	BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903 )	5 years ago
event_helpers.h	BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903 )	5 years ago
experimental.cc	Organizing rocksdb/db directory	6 years ago
external_sst_file_basic_test.cc	Allow ingesting overlapping files (#5539 )	5 years ago
external_sst_file_ingestion_job.cc	BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903 )	5 years ago
external_sst_file_ingestion_job.h	Allow ingesting overlapping files (#5539 )	5 years ago
external_sst_file_test.cc	Fix a bug in file ingestion (#5760 )	5 years ago
fault_injection_test.cc	Move some logging related files to logging/ (#5387 )	6 years ago
file_indexer.cc	Change RocksDB License	7 years ago
file_indexer.h	Move some memory related files from util/ to memory/ (#5382 )	6 years ago
file_indexer_test.cc	Move some memory related files from util/ to memory/ (#5382 )	6 years ago
filename_test.cc	Move some logging related files to logging/ (#5387 )	6 years ago
flush_job.cc	Propagate SST and blob file numbers through the EventListener interface (#5962 )	5 years ago
flush_job.h	Fix OnFlushCompleted fired before flush result write to MANIFEST (#5908 )	5 years ago
flush_job_test.cc	Fix OnFlushCompleted fired before flush result write to MANIFEST (#5908 )	5 years ago
flush_scheduler.cc	Refactor trimming logic for immutable memtables (#5022 )	5 years ago
flush_scheduler.h	Refactor trimming logic for immutable memtables (#5022 )	5 years ago
forward_iterator.cc	Add more callers for table reader. (#5454 )	5 years ago
forward_iterator.h	Move some memory related files from util/ to memory/ (#5382 )	6 years ago
forward_iterator_bench.cc	simplify include directive involving inttypes (#5402 )	6 years ago
import_column_family_job.cc	BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903 )	5 years ago
import_column_family_job.h	Apply formatter to recent 200+ commits. (#5830 )	5 years ago
import_column_family_test.cc	Apply formatter to recent 200+ commits. (#5830 )	5 years ago
internal_stats.cc	Apply formatter to recent 200+ commits. (#5830 )	5 years ago
internal_stats.h	Rename InternalDBStatsType enum names (#5779 )	5 years ago
job_context.h	WritePrepared: Fix visible key compacted out by compaction (#4883 )	6 years ago
listener_test.cc	Propagate SST and blob file numbers through the EventListener interface (#5962 )	5 years ago
log_format.h	Fix an inaccurate comment (#4315 )	6 years ago
log_reader.cc	Divide file_reader_writer.h and .cc (#5803 )	5 years ago
log_reader.h	Divide file_reader_writer.h and .cc (#5803 )	5 years ago
log_test.cc	Divide file_reader_writer.h and .cc (#5803 )	5 years ago
log_writer.cc	Divide file_reader_writer.h and .cc (#5803 )	5 years ago
log_writer.h	Use delete to disable automatic generated methods. (#5009 )	5 years ago
logs_with_prep_tracker.cc	Skip deleted WALs during recovery	7 years ago
logs_with_prep_tracker.h	Skip deleted WALs during recovery	7 years ago
lookup_key.h	Avoid user key copying for Get/Put/Write with user-timestamp (#5502 )	5 years ago
malloc_stats.cc	Support jemalloc compiled with `--with-jemalloc-prefix` (#5521 )	5 years ago
malloc_stats.h	Change RocksDB License	7 years ago
manual_compaction_test.cc	Move some memory related files from util/ to memory/ (#5382 )	6 years ago
memtable.cc	Misc hashing updates / upgrades (#5909 )	5 years ago
memtable.h	Fix OnFlushCompleted fired before flush result write to MANIFEST (#5908 )	5 years ago
memtable_list.cc	Fix OnFlushCompleted fired before flush result write to MANIFEST (#5908 )	5 years ago
memtable_list.h	Fix OnFlushCompleted fired before flush result write to MANIFEST (#5908 )	5 years ago
memtable_list_test.cc	Fix OnFlushCompleted fired before flush result write to MANIFEST (#5908 )	5 years ago
merge_context.h	Introduce a new MultiGet batching implementation (#5011 )	6 years ago
merge_helper.cc	Fix merging range tombstone covering put during flush/compaction (#5406 )	6 years ago
merge_helper.h	Remove v1 RangeDelAggregator (#4778 )	6 years ago
merge_helper_test.cc	Move some memory related files from util/ to memory/ (#5382 )	6 years ago
merge_operator.cc	Change RocksDB License	7 years ago
merge_test.cc	Make format	6 years ago
obsolete_files_test.cc	Refactor ObsoleteFilesTest to inherit from DBTestBase (#5820 )	5 years ago
options_file_test.cc	Organizing rocksdb/db directory	6 years ago
perf_context_test.cc	Move some memory related files from util/ to memory/ (#5382 )	6 years ago
pinned_iterators_manager.h	Change RocksDB License	7 years ago
plain_table_db_test.cc	Fix memory leak on error opening PlainTable (#5951 )	5 years ago
pre_release_callback.h	WritePrepared: reduce prepared_mutex_ overhead (#5420 )	6 years ago
prefix_test.cc	Organizing rocksdb/db directory	6 years ago
range_del_aggregator.cc	Organizing rocksdb/db directory	6 years ago
range_del_aggregator.h	Organizing rocksdb/db directory	6 years ago
range_del_aggregator_bench.cc	Move some memory related files from util/ to memory/ (#5382 )	6 years ago
range_del_aggregator_test.cc	Move test related files under util/ to test_util/ (#5377 )	6 years ago
range_tombstone_fragmenter.cc	Apply formatter to recent 200+ commits. (#5830 )	5 years ago
range_tombstone_fragmenter.h	Initialized pinned_pos_ and pinned_seq_pos_ in FragmentedRangeTombstoneIterator (#5720 )	5 years ago
range_tombstone_fragmenter_test.cc	Move test related files under util/ to test_util/ (#5377 )	6 years ago
read_callback.h	WriteUnPrepared: improve read your own write functionality (#5573 )	5 years ago
repair.cc	BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903 )	5 years ago
repair_test.cc	Organizing rocksdb/db directory	6 years ago
snapshot_checker.h	WritePrepared: fix issue with snapshot released during compaction (#4858 )	6 years ago
snapshot_impl.cc	Change RocksDB License	7 years ago
snapshot_impl.h	Refresh snapshot list during long compactions (2nd attempt) (#5278 )	6 years ago
table_cache.cc	Apply formatter to recent 200+ commits. (#5830 )	5 years ago
table_cache.h	Apply formatter to recent 200+ commits. (#5830 )	5 years ago
table_properties_collector.cc	Feature for sampling and reporting compressibility (#4842 )	6 years ago
table_properties_collector.h	Feature for sampling and reporting compressibility (#4842 )	6 years ago
table_properties_collector_test.cc	Divide file_reader_writer.h and .cc (#5803 )	5 years ago
transaction_log_impl.cc	Divide file_reader_writer.h and .cc (#5803 )	5 years ago
transaction_log_impl.h	reuse scratch buffer in transaction_log_reader (#5702 )	5 years ago
trim_history_scheduler.cc	Refactor trimming logic for immutable memtables (#5022 )	5 years ago
trim_history_scheduler.h	Refactor trimming logic for immutable memtables (#5022 )	5 years ago
version_builder.cc	save a few redundant container lookups (#5875 )	5 years ago
version_builder.h	Lower the risk for users to run options.force_consistency_checks = true (#5744 )	5 years ago
version_builder_test.cc	BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903 )	5 years ago
version_edit.cc	BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903 )	5 years ago
version_edit.h	BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903 )	5 years ago
version_edit_test.cc	BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903 )	5 years ago
version_set.cc	Update column families' log number altogether after flushing during recovery (#5856 )	5 years ago
version_set.h	Use delete to disable automatic generated methods. (#5009 )	5 years ago
version_set_test.cc	BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903 )	5 years ago
wal_manager.cc	Apply formatter to recent 200+ commits. (#5830 )	5 years ago
wal_manager.h	Adding DB::GetCurrentWalFile() API as a repliction/backup helper (#5765 )	5 years ago
wal_manager_test.cc	Apply formatter to recent 200+ commits. (#5830 )	5 years ago
write_batch.cc	Apply formatter to recent 200+ commits. (#5830 )	5 years ago
write_batch_base.cc	Change RocksDB License	7 years ago
write_batch_internal.h	Add insert hints for each writebatch (#5728 )	5 years ago
write_batch_test.cc	upgrade gtest 1.7.0 => 1.8.1 for json result writing	5 years ago
write_callback.h	Change RocksDB License	7 years ago
write_callback_test.cc	WritePrepared: reduce prepared_mutex_ overhead (#5420 )	6 years ago
write_controller.cc	Change RocksDB License	7 years ago
write_controller.h	Change RocksDB License	7 years ago
write_controller_test.cc	Move test related files under util/ to test_util/ (#5377 )	6 years ago
write_thread.cc	Option to make write group size configurable (#5759 )	5 years ago
write_thread.h	Option to make write group size configurable (#5759 )	5 years ago