fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Siying Dong d59549298f Skip deleted WALs during recovery ... Summary: This patch record min log number to keep to the manifest while flushing SST files to ignore them and any WAL older than them during recovery. This is to avoid scenarios when we have a gap between the WAL files are fed to the recovery procedure. The gap could happen by for example out-of-order WAL deletion. Such gap could cause problems in 2PC recovery where the prepared and commit entry are placed into two separate WAL and gap in the WALs could result into not processing the WAL with the commit entry and hence breaking the 2PC recovery logic. Before the commit, for 2PC case, we determined which log number to keep in FindObsoleteFiles(). We looked at the earliest logs with outstanding prepare entries, or prepare entries whose respective commit or abort are in memtable. With the commit, the same calculation is done while we apply the SST flush. Just before installing the flush file, we precompute the earliest log file to keep after the flush finishes using the same logic (but skipping the memtables just flushed), record this information to the manifest entry for this new flushed SST file. This pre-computed value is also remembered in memory, and will later be used to determine whether a log file can be deleted. This value is unlikely to change until next flush because the commit entry will stay in memtable. (In WritePrepared, we could have removed the older log files as soon as all prepared entries are committed. It's not yet done anyway. Even if we do it, the only thing we loss with this new approach is earlier log deletion between two flushes, which does not guarantee to happen anyway because the obsolete file clean-up function is only executed after flush or compaction) This min log number to keep is stored in the manifest using the safely-ignore customized field of AddFile entry, in order to guarantee that the DB generated using newer release can be opened by previous releases no older than 4.2. Closes https://github.com/facebook/rocksdb/pull/3765 Differential Revision: D7747618 Pulled By: siying fbshipit-source-id: d00c92105b4f83852e9754a1b70d6b64cb590729		7 years ago
..
builder.cc	Support for Column family specific paths.	7 years ago
builder.h	Stream	7 years ago
c.cc	expose WAL iterator in the C API	7 years ago
c_test.c	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
column_family.cc	Skip deleted WALs during recovery	7 years ago
column_family.h	Add max_subcompactions as a compaction option	7 years ago
column_family_test.cc	fix memory leak in two_level_iterator	7 years ago
compact_files_test.cc	fix memory leak in two_level_iterator	7 years ago
compacted_db_impl.cc	Fix coverity issues column_family, compaction_db/iterator	7 years ago
compacted_db_impl.h	Comment out unused variables	7 years ago
compaction.cc	Add max_subcompactions as a compaction option	7 years ago
compaction.h	Add max_subcompactions as a compaction option	7 years ago
compaction_iteration_stats.h	add counter for deletion dropping optimization	7 years ago
compaction_iterator.cc	Blob DB: Improve FIFO eviction	7 years ago
compaction_iterator.h	Comment out unused variables	7 years ago
compaction_iterator_test.cc	Comment out unused variables	7 years ago
compaction_job.cc	Add max_subcompactions as a compaction option	7 years ago
compaction_job.h	Make DBOption compaction_readahead_size dynamic	7 years ago
compaction_job_stats_test.cc	fix memory leak in two_level_iterator	7 years ago
compaction_job_test.cc	Add max_subcompactions as a compaction option	7 years ago
compaction_picker.cc	Add max_subcompactions as a compaction option	7 years ago
compaction_picker.h	Add max_subcompactions as a compaction option	7 years ago
compaction_picker_test.cc	Support for Column family specific paths.	7 years ago
compaction_picker_universal.cc	Add max_subcompactions as a compaction option	7 years ago
compaction_picker_universal.h	Make Universal compaction options dynamic	7 years ago
comparator_db_test.cc	Comment out unused variables	7 years ago
convenience.cc	Delete files in multiple ranges at once	7 years ago
corruption_test.cc	fix memory leak in two_level_iterator	7 years ago
cuckoo_table_db_test.cc	fix memory leak in two_level_iterator	7 years ago
db_basic_test.cc	Disallow compactions if there isn't enough free space	7 years ago
db_blob_index_test.cc	fix lite build	7 years ago
db_block_cache_test.cc	avoid double delete on dummy record insertion failure	7 years ago
db_bloom_filter_test.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
db_compaction_filter_test.cc	fix wrong length in snprintf	7 years ago
db_compaction_test.cc	fix memory leak in two_level_iterator	7 years ago
db_dynamic_level_test.cc	fix memory leak in two_level_iterator	7 years ago
db_encryption_test.cc	Change RocksDB License	7 years ago
db_filesnapshot.cc	fix behavior does not match name for "IsFileDeletionsEnabled"	7 years ago
db_flush_test.cc	Skip deleted WALs during recovery	7 years ago
db_impl.cc	Skip deleted WALs during recovery	7 years ago
db_impl.h	Skip deleted WALs during recovery	7 years ago
db_impl_compaction_flush.cc	Skip deleted WALs during recovery	7 years ago
db_impl_debug.cc	Skip deleted WALs during recovery	7 years ago
db_impl_experimental.cc	Inform caller when rocksdb is stalling writes	7 years ago
db_impl_files.cc	Skip deleted WALs during recovery	7 years ago
db_impl_open.cc	Skip deleted WALs during recovery	7 years ago
db_impl_readonly.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
db_impl_readonly.h	allowing CompactFiles to return new file names	7 years ago
db_impl_write.cc	Skip deleted WALs during recovery	7 years ago
db_info_dumper.cc	Change RocksDB License	7 years ago
db_info_dumper.h	Change RocksDB License	7 years ago
db_inplace_update_test.cc	Change RocksDB License	7 years ago
db_io_failure_test.cc	Fix LITE unit tests	7 years ago
db_iter.cc	Add rocksdb.iterator.internal-key property	7 years ago
db_iter.h	WritePrepared Txn: Return NotSupported on iterator refresh	7 years ago
db_iter_test.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
db_iterator_test.cc	Comment out unused variables	7 years ago
db_log_iter_test.cc	fix memory leak in two_level_iterator	7 years ago
db_memtable_test.cc	Comment out unused variables	7 years ago
db_merge_operator_test.cc	Fix TSAN timeout in MergeOperatorPinningTest.Randomized/x test	7 years ago
db_options_test.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
db_properties_test.cc	Add block cache related DB properties	7 years ago
db_range_del_test.cc	Add max_subcompactions as a compaction option	7 years ago
db_sst_test.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
db_statistics_test.cc	Change RocksDB License	7 years ago
db_table_properties_test.cc	fix deletion-triggered compaction in table builder	7 years ago
db_tailing_iter_test.cc	fix memory leak in two_level_iterator	7 years ago
db_test.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
db_test2.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
db_test_util.cc	Support for Column family specific paths.	7 years ago
db_test_util.h	Skip deleted WALs during recovery	7 years ago
db_universal_compaction_test.cc	fix memory leak in two_level_iterator	7 years ago
db_wal_test.cc	Skip deleted WALs during recovery	7 years ago
db_write_test.cc	Rename function for handling WAL write error	7 years ago
dbformat.cc	add kEntryRangeDeletion	7 years ago
dbformat.h	Imporve perf of random read and insert compare by suggesting inlining to the compiler	7 years ago
dbformat_test.cc	Change RocksDB License	7 years ago
deletefile_test.cc	fix memory leak in two_level_iterator	7 years ago
event_helpers.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
event_helpers.h	Change RocksDB License	7 years ago
experimental.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
external_sst_file_basic_test.cc	optimize file ingestion checks for range deletion overlap	7 years ago
external_sst_file_ingestion_job.cc	Improve accuracy of I/O stats collection of external SST ingestion.	7 years ago
external_sst_file_ingestion_job.h	Initialize a boolean member variable of a struct.	7 years ago
external_sst_file_test.cc	Skip deleted WALs during recovery	7 years ago
fault_injection_test.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
file_indexer.cc	Change RocksDB License	7 years ago
file_indexer.h	Change RocksDB License	7 years ago
file_indexer_test.cc	Comment out unused variables	7 years ago
filename_test.cc	Change RocksDB License	7 years ago
flush_job.cc	Skip deleted WALs during recovery	7 years ago
flush_job.h	Skip deleted WALs during recovery	7 years ago
flush_job_test.cc	Skip deleted WALs during recovery	7 years ago
flush_scheduler.cc	Change RocksDB License	7 years ago
flush_scheduler.h	Change RocksDB License	7 years ago
forward_iterator.cc	Comment out unused variables	7 years ago
forward_iterator.h	Comment out unused variables	7 years ago
forward_iterator_bench.cc	fix gflags namespace	7 years ago
internal_stats.cc	Add block cache related DB properties	7 years ago
internal_stats.h	Skip deleted WALs during recovery	7 years ago
job_context.h	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
listener_test.cc	fix some text in comments.	7 years ago
log_format.h	Change RocksDB License	7 years ago
log_reader.cc	Change RocksDB License	7 years ago
log_reader.h	Suppress lint in old files	7 years ago
log_test.cc	Use nullptr instead of NULL / 0 more consistently.	7 years ago
log_writer.cc	check return status for Sync() and Append() calls to avoid corruption	7 years ago
log_writer.h	Clarification regarding record format	7 years ago
logs_with_prep_tracker.cc	Skip deleted WALs during recovery	7 years ago
logs_with_prep_tracker.h	Skip deleted WALs during recovery	7 years ago
malloc_stats.cc	Disallow compactions if there isn't enough free space	7 years ago
malloc_stats.h	Change RocksDB License	7 years ago
managed_iterator.cc	Change RocksDB License	7 years ago
managed_iterator.h	Change RocksDB License	7 years ago
manual_compaction_test.cc	Comment out unused variables	7 years ago
memtable.cc	WritePrepared Txn: add stats	7 years ago
memtable.h	InlineSkiplist: don't decode keys unnecessarily during comparisons	7 years ago
memtable_list.cc	Skip deleted WALs during recovery	7 years ago
memtable_list.h	Skip deleted WALs during recovery	7 years ago
memtable_list_test.cc	Skip deleted WALs during recovery	7 years ago
merge_context.h	Change RocksDB License	7 years ago
merge_helper.cc	WritePrepared Txn: Support merge operator	7 years ago
merge_helper.h	WritePrepared Txn: Support merge operator	7 years ago
merge_helper_test.cc	Change RocksDB License	7 years ago
merge_operator.cc	Change RocksDB License	7 years ago
merge_test.cc	Comment out unused variables	7 years ago
obsolete_files_test.cc	fix memory leak in two_level_iterator	7 years ago
options_file_test.cc	fix memory leak in two_level_iterator	7 years ago
perf_context_test.cc	Improve write time breakdown stats	7 years ago
pinned_iterators_manager.h	Change RocksDB License	7 years ago
plain_table_db_test.cc	fix memory leak in two_level_iterator	7 years ago
pre_release_callback.h	Fix pre_release callback argument list.	7 years ago
prefix_test.cc	fix memory leak in two_level_iterator	7 years ago
range_del_aggregator.cc	Revert "Avoid adding tombstones of the same file to RangeDelAggregato…	7 years ago
range_del_aggregator.h	Revert "Avoid adding tombstones of the same file to RangeDelAggregato…	7 years ago
range_del_aggregator_test.cc	optimize file ingestion checks for range deletion overlap	7 years ago
read_callback.h	write-prepared txn: call IsInSnapshot	7 years ago
repair.cc	Support for Column family specific paths.	7 years ago
repair_test.cc	fix memory leak in two_level_iterator	7 years ago
snapshot_checker.h	Comment out unused variables	7 years ago
snapshot_impl.cc	Change RocksDB License	7 years ago
snapshot_impl.h	WritePrepared Txn: smallest_prepare optimization	7 years ago
table_cache.cc	Support for Column family specific paths.	7 years ago
table_cache.h	Change RocksDB License	7 years ago
table_properties_collector.cc	Comment out unused variables	7 years ago
table_properties_collector.h	Comment out unused variables	7 years ago
table_properties_collector_test.cc	Comment out unused variables	7 years ago
transaction_log_impl.cc	Comment out unused variables	7 years ago
transaction_log_impl.h	WritePrepared Txn: Refactor conf params	7 years ago
version_builder.cc	Fix some typos in comments and docs.	7 years ago
version_builder.h	Allow DB reopen with reduced options.num_levels	7 years ago
version_builder_test.cc	Change RocksDB License	7 years ago
version_edit.cc	Skip deleted WALs during recovery	7 years ago
version_edit.h	Skip deleted WALs during recovery	7 years ago
version_edit_test.cc	Skip deleted WALs during recovery	7 years ago
version_set.cc	Skip deleted WALs during recovery	7 years ago
version_set.h	Skip deleted WALs during recovery	7 years ago
version_set_test.cc	Comment out unused variables	7 years ago
wal_manager.cc	Fix memleak when DB::DeleteFile()	7 years ago
wal_manager.h	Fix memleak when DB::DeleteFile()	7 years ago
wal_manager_test.cc	fix memory leak in two_level_iterator	7 years ago
write_batch.cc	WritePrepared Txn: enable TryAgain for duplicates at the end of the batch	7 years ago
write_batch_base.cc	Change RocksDB License	7 years ago
write_batch_internal.h	Fix some typos in comments and docs.	7 years ago
write_batch_test.cc	Comment out unused variables	7 years ago
write_callback.h	Change RocksDB License	7 years ago
write_callback_test.cc	fix memory leak in two_level_iterator	7 years ago
write_controller.cc	Change RocksDB License	7 years ago
write_controller.h	Change RocksDB License	7 years ago
write_controller_test.cc	Change RocksDB License	7 years ago
write_thread.cc	Improve write time breakdown stats	7 years ago
write_thread.h	WritePrepared Txn: Duplicate Keys, Txn Part	7 years ago