rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

History

Akanksha Mahajan ae82d91492 Remove corrupted WAL files in kPointRecoveryMode with avoid_flush_duing_recovery set true (#9634 ) Summary: 1) In case of non-TransactionDB and avoid_flush_during_recovery = true, RocksDB won't flush the data from WAL to L0 for all column families if possible. As a result, not all column families can increase their log_numbers, and min_log_number_to_keep won't change. 2) For transaction DB (.allow_2pc), even with the flush, there may be old WAL files that it must not delete because they can contain data of uncommitted transactions and min_log_number_to_keep won't change. If we persist a new MANIFEST with advanced log_numbers for some column families, then during a second crash after persisting the MANIFEST, RocksDB will see some column families' log_numbers larger than the corrupted wal, and the "column family inconsistency" error will be hit, causing recovery to fail. As a solution, 1. the corrupted WALs whose numbers are larger than the corrupted wal and smaller than the new WAL will be moved to archive folder. 2. Currently, RocksDB DB::Open() may creates and writes to two new MANIFEST files even before recovery succeeds. This PR buffers the edits in a structure and writes to a new MANIFEST after recovery is successful Pull Request resolved: https://github.com/facebook/rocksdb/pull/9634 Test Plan: 1. Added new unit tests 2. make crast_test -j Reviewed By: riversand963 Differential Revision: D34463666 Pulled By: akankshamahajan15 fbshipit-source-id: e233d3af0ed4e2028ca0cf051e5a334a0fdc9d19		3 years ago
..
blob	Add rate limiter priority to ReadOptions (#9424 )	3 years ago
compaction	Remove dead code (#9825 )	3 years ago
db_impl	Remove corrupted WAL files in kPointRecoveryMode with avoid_flush_duing_recovery set true (#9634 )	3 years ago
arena_wrapped_db_iter.cc	fix: Reusing-Iterator reads stale keys after DeleteRange() performed (#9258 )	3 years ago
arena_wrapped_db_iter.h	Cleanup includes in dbformat.h (#8930 )	3 years ago
builder.cc	Remove dead code (#9825 )	3 years ago
builder.h	Expose blob file information through the EventListener interface (#8675 )	3 years ago
c.cc	Fix a major performance bug in 7.0 re: filter compatibility (#9736 )	3 years ago
c_test.c	Fix remaining uses of "backupable" (#9792 )	3 years ago
column_family.cc	Fix heap use-after-free race with DropColumnFamily (#9730 )	3 years ago
column_family.h	Fix heap use-after-free race with DropColumnFamily (#9730 )	3 years ago
column_family_test.cc	More refactoring ahead of footer & meta changes (#9240 )	3 years ago
compact_files_test.cc	Fix test race conditions with OnFlushCompleted() (#9617 )	3 years ago
comparator_db_test.cc	More refactoring ahead of footer & meta changes (#9240 )	3 years ago
convenience.cc	Add rate limiter priority to ReadOptions (#9424 )	3 years ago
corruption_test.cc	Remove corrupted WAL files in kPointRecoveryMode with avoid_flush_duing_recovery set true (#9634 )	3 years ago
cuckoo_table_db_test.cc	Experimental support for SST unique IDs (#8990 )	3 years ago
db_basic_test.cc	Remove corrupted WAL files in kPointRecoveryMode with avoid_flush_duing_recovery set true (#9634 )	3 years ago
db_block_cache_test.cc	Enhance new cache key testing & comments (#9329 )	3 years ago
db_bloom_filter_test.cc	Tests for filter compatibility (#9773 )	3 years ago
db_compaction_filter_test.cc	Fix a minor issue with initializing the test path (#8555 )	3 years ago
db_compaction_test.cc	L0 Subcompaction to trim input files (#9802 )	3 years ago
db_dynamic_level_test.cc	Remove deprecated API AdvancedColumnFamilyOptions::soft_rate_limit/hard_rate_limit (#9452 )	3 years ago
db_encryption_test.cc	Fix a minor issue with initializing the test path (#8555 )	3 years ago
db_filesnapshot.cc	Improve comments for some files (#9793 )	3 years ago
db_flush_test.cc	Encode min_log_number_to_keep and delete_wals_before in one version edit (#9766 )	3 years ago
db_info_dumper.cc	Allow WAL dir to change with db dir (#8582 )	3 years ago
db_info_dumper.h	Add a DB Session ID (#6959 )	4 years ago
db_inplace_update_test.cc	fix a bug, c api, if enable inplace_update_support, and use create sn… (#9471 )	3 years ago
db_io_failure_test.cc	Enable a few unit tests to use custom Env objects (#9087 )	3 years ago
db_iter.cc	Remove dead code (#9825 )	3 years ago
db_iter.h	Remove dead code (#9825 )	3 years ago
db_iter_stress_test.cc	Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262 )	4 years ago
db_iter_test.cc	Remove iter_start_seqnum and preserve_deletes (#9430 )	3 years ago
db_iterator_test.cc	Fix a minor issue with initializing the test path (#8555 )	3 years ago
db_kv_checksum_test.cc	Revise APIs related to user-defined timestamp (#8946 )	3 years ago
db_log_iter_test.cc	Attempt to deflake DBTestXactLogIterator.TransactionLogIteratorCorruptedLog (#8627 )	3 years ago
db_logical_block_size_cache_test.cc	Attempt to deflake DBLogicalBlockSizeCacheTest.CreateColumnFamilies (#9516 )	3 years ago
db_memtable_test.cc	Fix a minor issue with initializing the test path (#8555 )	3 years ago
db_merge_operand_test.cc	Fix GetMergeOperands() heap-use-after-free on flushed memtable (#9805 )	3 years ago
db_merge_operator_test.cc	Fix a minor issue with initializing the test path (#8555 )	3 years ago
db_options_test.cc	Remove deprecated option new_table_reader_for_compaction_inputs (#9443 )	3 years ago
db_properties_test.cc	Tests for filter compatibility (#9773 )	3 years ago
db_range_del_test.cc	fix: Reusing-Iterator reads stale keys after DeleteRange() performed (#9258 )	3 years ago
db_rate_limiter_test.cc	Rate-limit automatic WAL flush after each user write (#9607 )	3 years ago
db_secondary_test.cc	Make the Env class Customizable (#9293 )	3 years ago
db_sst_test.cc	Account memory of big memory users in BlockBasedTable in global memory limit (#9748 )	3 years ago
db_statistics_test.cc	Bytes read stat for `VerifyChecksum()` and `VerifyFileChecksums()` APIs (#8741 )	3 years ago
db_table_properties_test.cc	Account memory of big memory users in BlockBasedTable in global memory limit (#9748 )	3 years ago
db_tailing_iter_test.cc	Fix a minor issue with initializing the test path (#8555 )	3 years ago
db_test.cc	Fix FileStorageInfo fields from GetLiveFilesMetaData (#9769 )	3 years ago
db_test2.cc	Add manifest fix-up utility for file temperatures (#9683 )	3 years ago
db_test_util.cc	Use a sorted vector instead of a map to store blob file metadata (#9526 )	3 years ago
db_test_util.h	Update Cache::Release param from force_erase to erase_if_last_ref (#9728 )	3 years ago
db_universal_compaction_test.cc	Adhere to per-DB concurrency limit when bottom-pri compactions exist (#9179 )	3 years ago
db_wal_test.cc	Remove corrupted WAL files in kPointRecoveryMode with avoid_flush_duing_recovery set true (#9634 )	3 years ago
db_with_timestamp_basic_test.cc	Add OpenAndTrimHistory API to support trimming data with specified timestamp (#9410 )	3 years ago
db_with_timestamp_compaction_test.cc	Use the comparator from the sst file table properties in sst_dump_tool (#9491 )	3 years ago
db_write_buffer_manager_test.cc	Enable a few unit tests to use custom Env objects (#9087 )	3 years ago
db_write_test.cc	Enable a few unit tests to use custom Env objects (#9087 )	3 years ago
dbformat.cc	Track per-SST user-defined timestamp information in MANIFEST (#9092 )	3 years ago
dbformat.h	Add OpenAndTrimHistory API to support trimming data with specified timestamp (#9410 )	3 years ago
dbformat_test.cc	Enable a few unit tests to use custom Env objects (#9087 )	3 years ago
deletefile_test.cc	Enable a few unit tests to use custom Env objects (#9087 )	3 years ago
error_handler.cc	Expand auto recovery to background read errors (#9679 )	3 years ago
error_handler.h	Expand auto recovery to background read errors (#9679 )	3 years ago
error_handler_fs_test.cc	Expand auto recovery to background read errors (#9679 )	3 years ago
event_helpers.cc	Fix a race condition in WAL tracking causing DB open failure (#9715 )	3 years ago
event_helpers.h	Add a listener callback for end of auto error recovery (#9244 )	3 years ago
experimental.cc	Add manifest fix-up utility for file temperatures (#9683 )	3 years ago
external_sst_file_basic_test.cc	Enable a few unit tests to use custom Env objects (#9087 )	3 years ago
external_sst_file_ingestion_job.cc	Do not rely on ADL when invoking std::max_element (#9608 )	3 years ago
external_sst_file_ingestion_job.h	New stable, fixed-length cache keys (#9126 )	3 years ago
external_sst_file_test.cc	Make the Env class Customizable (#9293 )	3 years ago
fault_injection_test.cc	Fix a bug causing duplicate trailing entries in WritableFile (buffered IO) (#9236 )	3 years ago
file_indexer.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
file_indexer.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
file_indexer_test.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
filename_test.cc	fixing issue #8345 RocksDB does not work when using UNC network paths (#9384 )	3 years ago
flush_job.cc	Remove dead code (#9825 )	3 years ago
flush_job.h	Expand auto recovery to background read errors (#9679 )	3 years ago
flush_job_test.cc	Use the comparator from the sst file table properties in sst_dump_tool (#9491 )	3 years ago
flush_scheduler.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
flush_scheduler.h	Include C++ standard library headers instead of C compatibility headers (#8068 )	4 years ago
forward_iterator.cc	Fast path for detecting unchanged prefix_extractor (#9407 )	3 years ago
forward_iterator.h	Fast path for detecting unchanged prefix_extractor (#9407 )	3 years ago
forward_iterator_bench.cc	Remove using namespace (#9369 )	3 years ago
history_trimming_iterator.h	Add OpenAndTrimHistory API to support trimming data with specified timestamp (#9410 )	3 years ago
import_column_family_job.cc	Add Temperature info in `NewSequentialFile()` (#9499 )	3 years ago
import_column_family_job.h	New stable, fixed-length cache keys (#9126 )	3 years ago
import_column_family_test.cc	Fix a minor issue with initializing the test path (#8555 )	3 years ago
internal_stats.cc	Add manifest fix-up utility for file temperatures (#9683 )	3 years ago
internal_stats.h	Support `GetMapProperty()` with "rocksdb.dbstats" (#9057 )	3 years ago
job_context.h	Add manifest fix-up utility for file temperatures (#9683 )	3 years ago
kv_checksum.h	fix compile errors in db/kv_checksum.h (#9173 )	3 years ago
listener_test.cc	Fix test race conditions with OnFlushCompleted() (#9617 )	3 years ago
log_format.h	Add record to set WAL compression type if enabled (#9556 )	3 years ago
log_reader.cc	Integrate WAL compression into log reader/writer. (#9642 )	3 years ago
log_reader.h	Integrate WAL compression into log reader/writer. (#9642 )	3 years ago
log_test.cc	Integrate WAL compression into log reader/writer. (#9642 )	3 years ago
log_writer.cc	Integrate WAL compression into log reader/writer. (#9642 )	3 years ago
log_writer.h	Integrate WAL compression into log reader/writer. (#9642 )	3 years ago
logs_with_prep_tracker.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
logs_with_prep_tracker.h	Include C++ standard library headers instead of C compatibility headers (#8068 )	4 years ago
lookup_key.h	Cleanup includes in dbformat.h (#8930 )	3 years ago
malloc_stats.cc	Replace most typedef with using= (#8751 )	3 years ago
malloc_stats.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
manual_compaction_test.cc	Remove using namespace (#9369 )	3 years ago
memtable.cc	Fix major bug with MultiGet, DeleteRange, and memtable Bloom (#9453 )	3 years ago
memtable.h	Fix major bug with MultiGet, DeleteRange, and memtable Bloom (#9453 )	3 years ago
memtable_list.cc	Encode min_log_number_to_keep and delete_wals_before in one version edit (#9766 )	3 years ago
memtable_list.h	Expand auto recovery to background read errors (#9679 )	3 years ago
memtable_list_test.cc	Expand auto recovery to background read errors (#9679 )	3 years ago
merge_context.h	Add Merge Operator support to WriteBatchWithIndex (#8135 )	4 years ago
merge_helper.cc	Support readahead during compaction for blob files (#9187 )	3 years ago
merge_helper.h	Support readahead during compaction for blob files (#9187 )	3 years ago
merge_helper_test.cc	Support readahead during compaction for blob files (#9187 )	3 years ago
merge_operator.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
merge_test.cc	Make the Env class Customizable (#9293 )	3 years ago
obsolete_files_test.cc	Add commit marker with timestamp (#9266 )	3 years ago
options_file_test.cc	No elide constructors (#7798 )	4 years ago
output_validator.cc	Cleanup includes in dbformat.h (#8930 )	3 years ago
output_validator.h	Cleanup includes in dbformat.h (#8930 )	3 years ago
perf_context_test.cc	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033 )	4 years ago
periodic_work_scheduler.cc	Fix a timer crash caused by invalid memory management (#9656 )	3 years ago
periodic_work_scheduler.h	Fix a timer crash caused by invalid memory management (#9656 )	3 years ago
periodic_work_scheduler_test.cc	Fix a minor issue with initializing the test path (#8555 )	3 years ago
pinned_iterators_manager.h	Replace most typedef with using= (#8751 )	3 years ago
plain_table_db_test.cc	Fast path for detecting unchanged prefix_extractor (#9407 )	3 years ago
pre_release_callback.h	Fix and detect headers with missing dependencies (#8893 )	3 years ago
prefix_test.cc	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033 )	4 years ago
range_del_aggregator.cc	In ParseInternalKey(), include corrupt key info in Status (#7515 )	4 years ago
range_del_aggregator.h	Fix some typos in comments (#8066 )	4 years ago
range_del_aggregator_bench.cc	Cleanup multiple implementations of VectorIterator (#8901 )	3 years ago
range_del_aggregator_test.cc	Cleanup multiple implementations of VectorIterator (#8901 )	3 years ago
range_tombstone_fragmenter.cc	Added memtable garbage statistics (#8411 )	3 years ago
range_tombstone_fragmenter.h	Added memtable garbage statistics (#8411 )	3 years ago
range_tombstone_fragmenter_test.cc	Cleanup multiple implementations of VectorIterator (#8901 )	3 years ago
read_callback.h	Fix and detect headers with missing dependencies (#8893 )	3 years ago
repair.cc	Fast path for detecting unchanged prefix_extractor (#9407 )	3 years ago
repair_test.cc	Some fixes and enhancements to `ldb repair` (#8544 )	3 years ago
snapshot_checker.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
snapshot_impl.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
snapshot_impl.h	Fix and detect headers with missing dependencies (#8893 )	3 years ago
table_cache.cc	fix a bug of the ticker NO_FILE_OPENS (#9677 )	3 years ago
table_cache.h	Fast path for detecting unchanged prefix_extractor (#9407 )	3 years ago
table_properties_collector.cc	Apply `sample_for_compression` to all block-based tables (#8105 )	4 years ago
table_properties_collector.h	Track each SST's timestamp information as user properties (#9093 )	3 years ago
table_properties_collector_test.cc	Improve / clean up meta block code & integrity (#9163 )	3 years ago
transaction_log_impl.cc	Add commit marker with timestamp (#9266 )	3 years ago
transaction_log_impl.h	Cleanup includes in dbformat.h (#8930 )	3 years ago
trim_history_scheduler.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
trim_history_scheduler.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
version_builder.cc	Use a sorted vector instead of a map to store blob file metadata (#9526 )	3 years ago
version_builder.h	Fast path for detecting unchanged prefix_extractor (#9407 )	3 years ago
version_builder_test.cc	Use a sorted vector instead of a map to store blob file metadata (#9526 )	3 years ago
version_edit.cc	Encode min_log_number_to_keep and delete_wals_before in one version edit (#9766 )	3 years ago
version_edit.h	Clean up VersionStorageInfo a bit (#9494 )	3 years ago
version_edit_handler.cc	Fix a race condition in WAL tracking causing DB open failure (#9715 )	3 years ago
version_edit_handler.h	Fixed manifest_dump issues when printing keys and values containing null characters (#8378 )	3 years ago
version_edit_test.cc	File temperature information should be preserved when restart the DB (#9242 )	3 years ago
version_set.cc	L0 Subcompaction to trim input files (#9802 )	3 years ago
version_set.h	L0 Subcompaction to trim input files (#9802 )	3 years ago
version_set_test.cc	Fix a race condition in WAL tracking causing DB open failure (#9715 )	3 years ago
version_util.h	Add manifest fix-up utility for file temperatures (#9683 )	3 years ago
wal_edit.cc	Always track WAL obsoletion (#7759 )	4 years ago
wal_edit.h	Always track WAL obsoletion (#7759 )	4 years ago
wal_edit_test.cc	Always track WAL obsoletion (#7759 )	4 years ago
wal_manager.cc	Allow WAL dir to change with db dir (#8582 )	3 years ago
wal_manager.h	Allow WAL dir to change with db dir (#8582 )	3 years ago
wal_manager_test.cc	Make SystemClock into a Customizable Class (#8636 )	3 years ago
write_batch.cc	Fix a few documentation errors including in public APIs (#9789 )	3 years ago
write_batch_base.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
write_batch_internal.h	Support WBWI for keys having timestamps (#9603 )	3 years ago
write_batch_test.cc	Support WBWI for keys having timestamps (#9603 )	3 years ago
write_callback.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
write_callback_test.cc	Move slow valgrind tests behind -DROCKSDB_FULL_VALGRIND_RUN (#8475 )	3 years ago
write_controller.cc	Revamp WriteController (#8064 )	4 years ago
write_controller.h	Revamp WriteController (#8064 )	4 years ago
write_controller_test.cc	Revamp WriteController (#8064 )	4 years ago
write_thread.cc	Rate-limit automatic WAL flush after each user write (#9607 )	3 years ago
write_thread.h	Rate-limit automatic WAL flush after each user write (#9607 )	3 years ago