rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

History

Baptiste Lemaire 206845c057 Mempurge support for wal (#8528 ) Summary: In this PR, `mempurge` is made compatible with the Write Ahead Log: in case of recovery, the DB is now capable of recovering the data that was "mempurged" and kept in the `imm()` list of immutable memtables. The twist was to add a uint64_t to the `memtable` struct to store the number of the earliest log file containing entries from the `memtable`. When a `Flush` operation is replaced with a `MemPurge`, the `VersionEdit` (which usually contains the new min log file number to pick up for recovery and the level 0 file path of the newly created SST file) is no longer appended to the manifest log, and every time the `deleteWal` method is called, a check is made on the list of immutable memtables. This PR also includes a unit test that verifies that no data is lost upon Reopening of the database when the mempurge feature is activated. This extensive unit test includes two column families, with valid data contained in the imm() at time of "crash"/reopening (recovery). Pull Request resolved: https://github.com/facebook/rocksdb/pull/8528 Reviewed By: pdillinger Differential Revision: D29701097 Pulled By: bjlemaire fbshipit-source-id: 072a900fb6ccc1edcf5eef6caf88f3060238edf9		3 years ago
..
blob	Add BlobMetaData retrieval methods (#8273 )	3 years ago
compaction	Call OnCompactionCompleted API in case of DisableManualCompaction (#8469 )	3 years ago
db_impl	Mempurge support for wal (#8528 )	3 years ago
arena_wrapped_db_iter.cc	Rename ImmutableOptions variables (#8409 )	3 years ago
arena_wrapped_db_iter.h	Rename ImmutableOptions variables (#8409 )	3 years ago
builder.cc	Memtable "MemPurge" prototype (#8454 )	3 years ago
builder.h	Added memtable garbage statistics (#8411 )	3 years ago
c.cc	Add ribbon filter to C API (#8486 )	3 years ago
c_test.c	Add ribbon filter to C API (#8486 )	3 years ago
column_family.cc	Mempurge support for wal (#8528 )	3 years ago
column_family.h	Mempurge support for wal (#8528 )	3 years ago
column_family_test.cc	Add CreateFrom methods to Env/FileSystem (#8174 )	3 years ago
compact_files_test.cc	Compaction should not move data to up level (#8116 )	4 years ago
comparator_db_test.cc	Remove unused includes (#7604 )	4 years ago
convenience.cc	Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262 )	4 years ago
corruption_test.cc	Add CreateFrom methods to Env/FileSystem (#8174 )	3 years ago
cuckoo_table_db_test.cc	Revert "Turn on memtable bloom filter by default. (#6584 )" (#7939 )	4 years ago
db_basic_test.cc	Initial support for secondary cache in LRUCache (#8271 )	4 years ago
db_block_cache_test.cc	Fix Get() return status when block cache is disabled (#8485 )	3 years ago
db_bloom_filter_test.cc	Move slow valgrind tests behind -DROCKSDB_FULL_VALGRIND_RUN (#8475 )	3 years ago
db_compaction_filter_test.cc	Move slow valgrind tests behind -DROCKSDB_FULL_VALGRIND_RUN (#8475 )	3 years ago
db_compaction_test.cc	Move slow valgrind tests behind -DROCKSDB_FULL_VALGRIND_RUN (#8475 )	3 years ago
db_dynamic_level_test.cc	Add a SystemClock class to capture the time functions of an Env (#7858 )	4 years ago
db_encryption_test.cc	Improvements to Env::GetChildren (#7819 )	4 years ago
db_filesnapshot.cc	Fix `GetLiveFiles()` returning OPTIONS-000000 (#8268 )	4 years ago
db_flush_test.cc	Mempurge support for wal (#8528 )	3 years ago
db_info_dumper.cc	Fix write-ahead log file size overflow (#7870 )	4 years ago
db_info_dumper.h	Add a DB Session ID (#6959 )	4 years ago
db_inplace_update_test.cc	Whole DBTest to skip fsync (#7274 )	4 years ago
db_io_failure_test.cc	Add more tests to ASSERT_STATUS_CHECKED (3), API change (#7715 )	4 years ago
db_iter.cc	Rename ImmutableOptions variables (#8409 )	3 years ago
db_iter.h	Rename ImmutableOptions variables (#8409 )	3 years ago
db_iter_stress_test.cc	Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262 )	4 years ago
db_iter_test.cc	Rename ImmutableOptions variables (#8409 )	3 years ago
db_iterator_test.cc	Move slow valgrind tests behind -DROCKSDB_FULL_VALGRIND_RUN (#8475 )	3 years ago
db_kv_checksum_test.cc	Integrity protection for live updates to WriteBatch (#7748 )	4 years ago
db_log_iter_test.cc	Add a SystemClock class to capture the time functions of an Env (#7858 )	4 years ago
db_logical_block_size_cache_test.cc	Add further tests to ASSERT_STATUS_CHECKED (2) (#7698 )	4 years ago
db_memtable_test.cc	Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262 )	4 years ago
db_merge_operand_test.cc	Support for Merge in Integrated BlobDB with base values (#8292 )	3 years ago
db_merge_operator_test.cc	Add further tests to ASSERT_STATUS_CHECKED (1) (#7679 )	4 years ago
db_options_test.cc	Add ObjectRegistry to ConfigOptions (#8166 )	4 years ago
db_properties_test.cc	Assert unlimited max_open_files for FIFO compaction. (#8172 )	4 years ago
db_range_del_test.cc	Fix manual compaction `max_compaction_bytes` under-calculated issue (#8269 )	4 years ago
db_secondary_test.cc	Implement missing Handler methods in ColumnFamilyCollector. (#8456 )	3 years ago
db_sst_test.cc	Fix flaky failure in DBSSTest.DBWithSstFileManagerForBlobFilesWithGC (#8196 )	4 years ago
db_statistics_test.cc	Add more tests for assert status checked (#7524 )	4 years ago
db_table_properties_test.cc	Add more tests for assert status checked (#7524 )	4 years ago
db_tailing_iter_test.cc	Add more tests to ASSERT_STATUS_CHECKED (3), API change (#7715 )	4 years ago
db_test.cc	Move slow valgrind tests behind -DROCKSDB_FULL_VALGRIND_RUN (#8475 )	3 years ago
db_test2.cc	Call OnCompactionCompleted API in case of DisableManualCompaction (#8469 )	3 years ago
db_test_util.cc	Add CreateFrom methods to Env/FileSystem (#8174 )	3 years ago
db_test_util.h	Cache warming data blocks during flush (#8242 )	3 years ago
db_universal_compaction_test.cc	Move slow valgrind tests behind -DROCKSDB_FULL_VALGRIND_RUN (#8475 )	3 years ago
db_wal_test.cc	Use DbSessionId as cache key prefix when secondary cache is enabled (#8360 )	3 years ago
db_with_timestamp_basic_test.cc	Move slow valgrind tests behind -DROCKSDB_FULL_VALGRIND_RUN (#8475 )	3 years ago
db_with_timestamp_compaction_test.cc	Whole DBTest to skip fsync (#7274 )	4 years ago
db_write_buffer_manager_test.cc	Stall writes in WriteBufferManager when memory_usage exceeds buffer_size (#7898 )	4 years ago
db_write_test.cc	Add more tests to ASSERT_STATUS_CHECKED (4) (#7718 )	4 years ago
dbformat.cc	Make CompactRange and GetApproximateSizes work with timestamp (#7684 )	4 years ago
dbformat.h	Fix some typos in comments (#8066 )	4 years ago
dbformat_test.cc	Remove unused includes (#7604 )	4 years ago
deletefile_test.cc	Add more tests to ASSERT_STATUS_CHECKED (4) (#7718 )	4 years ago
error_handler.cc	All the NoSpace() errors will be handled by regular SetBGError and RecoverFromNoSpace() (#8376 )	3 years ago
error_handler.h	Fix some typos in comments (#8066 )	4 years ago
error_handler_fs_test.cc	All the NoSpace() errors will be handled by regular SetBGError and RecoverFromNoSpace() (#8376 )	3 years ago
event_helpers.cc	Add table properties for number of entries added to filters (#8323 )	4 years ago
event_helpers.h	Pass SST file checksum information through OnTableFileCreated (#7108 )	4 years ago
experimental.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
external_sst_file_basic_test.cc	Sync ingested files only if reopen is supported by the FS (#8296 )	4 years ago
external_sst_file_ingestion_job.cc	Sync ingested files only if reopen is supported by the FS (#8296 )	4 years ago
external_sst_file_ingestion_job.h	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033 )	4 years ago
external_sst_file_test.cc	Move slow valgrind tests behind -DROCKSDB_FULL_VALGRIND_RUN (#8475 )	3 years ago
fault_injection_test.cc	Add more tests for assert status checked (#7524 )	4 years ago
file_indexer.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
file_indexer.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
file_indexer_test.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
filename_test.cc	Remove unused includes (#7604 )	4 years ago
flush_job.cc	Mempurge support for wal (#8528 )	3 years ago
flush_job.h	Mempurge support for wal (#8528 )	3 years ago
flush_job_test.cc	Use DbSessionId as cache key prefix when secondary cache is enabled (#8360 )	3 years ago
flush_scheduler.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
flush_scheduler.h	Include C++ standard library headers instead of C compatibility headers (#8068 )	4 years ago
forward_iterator.cc	Fix some typos in comments (#8066 )	4 years ago
forward_iterator.h	Properly report IO errors when IndexType::kBinarySearchWithFirstKey is used (#6621 )	5 years ago
forward_iterator_bench.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
import_column_family_job.cc	Add a SystemClock class to capture the time functions of an Env (#7858 )	4 years ago
import_column_family_job.h	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033 )	4 years ago
import_column_family_test.cc	No elide constructors (#7798 )	4 years ago
internal_stats.cc	Pin CacheEntryStatsCollector to fix performance bug (#8385 )	3 years ago
internal_stats.h	Pin CacheEntryStatsCollector to fix performance bug (#8385 )	3 years ago
job_context.h	Rename ImmutableOptions variables (#8409 )	3 years ago
kv_checksum.h	Integrity protection for live updates to WriteBatch (#7748 )	4 years ago
listener_test.cc	Add more tests for assert status checked (#7524 )	4 years ago
log_format.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
log_reader.cc	Fix kPointInTimeRecovery handling of truncated WAL (#7701 )	4 years ago
log_reader.h	Real fix for race in backup custom checksum checking (#7309 )	4 years ago
log_test.cc	Make StringEnv, StringSink, StringSource use FS classes (#7786 )	4 years ago
log_writer.cc	Using existing crc32c checksum in checksum handoff for Manifest and WAL (#8412 )	3 years ago
log_writer.h	Include C++ standard library headers instead of C compatibility headers (#8068 )	4 years ago
logs_with_prep_tracker.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
logs_with_prep_tracker.h	Include C++ standard library headers instead of C compatibility headers (#8068 )	4 years ago
lookup_key.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
malloc_stats.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
malloc_stats.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
manual_compaction_test.cc	Add more tests for assert status checked (#7524 )	4 years ago
memtable.cc	Mempurge support for wal (#8528 )	3 years ago
memtable.h	Mempurge support for wal (#8528 )	3 years ago
memtable_list.cc	Mempurge support for wal (#8528 )	3 years ago
memtable_list.h	Mempurge support for wal (#8528 )	3 years ago
memtable_list_test.cc	Use DbSessionId as cache key prefix when secondary cache is enabled (#8360 )	3 years ago
merge_context.h	Add Merge Operator support to WriteBatchWithIndex (#8135 )	4 years ago
merge_helper.cc	Add support for Merge with base value during Compaction in IntegratedBlobDB (#8445 )	3 years ago
merge_helper.h	Add support for Merge with base value during Compaction in IntegratedBlobDB (#8445 )	3 years ago
merge_helper_test.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
merge_operator.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
merge_test.cc	MergeHelper::FilterMerge() calling ElapsedNanosSafe() upon exit even … (#7867 )	4 years ago
obsolete_files_test.cc	Print blob file checksums as hex (#8437 )	3 years ago
options_file_test.cc	No elide constructors (#7798 )	4 years ago
output_validator.cc	Use NPHash64 in more places (#7632 )	4 years ago
output_validator.h	Add remote compaction public API (#8300 )	4 years ago
perf_context_test.cc	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033 )	4 years ago
periodic_work_scheduler.cc	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033 )	4 years ago
periodic_work_scheduler.h	Add a SystemClock class to capture the time functions of an Env (#7858 )	4 years ago
periodic_work_scheduler_test.cc	Add a SystemClock class to capture the time functions of an Env (#7858 )	4 years ago
pinned_iterators_manager.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
plain_table_db_test.cc	Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262 )	4 years ago
pre_release_callback.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
prefix_test.cc	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033 )	4 years ago
range_del_aggregator.cc	In ParseInternalKey(), include corrupt key info in Status (#7515 )	4 years ago
range_del_aggregator.h	Fix some typos in comments (#8066 )	4 years ago
range_del_aggregator_bench.cc	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033 )	4 years ago
range_del_aggregator_test.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
range_tombstone_fragmenter.cc	Added memtable garbage statistics (#8411 )	3 years ago
range_tombstone_fragmenter.h	Added memtable garbage statistics (#8411 )	3 years ago
range_tombstone_fragmenter_test.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
read_callback.h	Get() with timestamp should respect snapshot (#7227 )	4 years ago
repair.cc	Mempurge support for wal (#8528 )	3 years ago
repair_test.cc	Use SST file manager to track blob files as well (#8037 )	4 years ago
snapshot_checker.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
snapshot_impl.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
snapshot_impl.h	Fix some typos in comments (#8066 )	4 years ago
table_cache.cc	Use DbSessionId as cache key prefix when secondary cache is enabled (#8360 )	3 years ago
table_cache.h	Use DbSessionId as cache key prefix when secondary cache is enabled (#8360 )	3 years ago
table_properties_collector.cc	Apply `sample_for_compression` to all block-based tables (#8105 )	4 years ago
table_properties_collector.h	Partially revert the "apply subrange of table property collectors" change (#8465 )	3 years ago
table_properties_collector_test.cc	Make it possible to apply only a subrange of table property collectors (#8298 )	4 years ago
transaction_log_impl.cc	Add more tests for assert status checked (#7524 )	4 years ago
transaction_log_impl.h	Store FileSystemPtr object that contains FileSystem ptr (#7180 )	4 years ago
trim_history_scheduler.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
trim_history_scheduler.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
version_builder.cc	Handle blob files when options.best_efforts_recovery is true (#8180 )	4 years ago
version_builder.h	Handle blob files when options.best_efforts_recovery is true (#8180 )	4 years ago
version_builder_test.cc	Print blob file checksums as hex (#8437 )	3 years ago
version_edit.cc	Write file temperature information to manifest (#8284 )	4 years ago
version_edit.h	Write file temperature information to manifest (#8284 )	4 years ago
version_edit_handler.cc	Mempurge support for wal (#8528 )	3 years ago
version_edit_handler.h	Fixed manifest_dump issues when printing keys and values containing null characters (#8378 )	3 years ago
version_edit_test.cc	Make it able to ignore WAL related VersionEdits in older versions (#7873 )	4 years ago
version_set.cc	Mempurge support for wal (#8528 )	3 years ago
version_set.h	Mempurge support for wal (#8528 )	3 years ago
version_set_test.cc	Print blob file checksums as hex (#8437 )	3 years ago
wal_edit.cc	Always track WAL obsoletion (#7759 )	4 years ago
wal_edit.h	Always track WAL obsoletion (#7759 )	4 years ago
wal_edit_test.cc	Always track WAL obsoletion (#7759 )	4 years ago
wal_manager.cc	Make types of Immutable/Mutable Options fields match that of the underlying Option (#8176 )	4 years ago
wal_manager.h	Store FileSystemPtr object that contains FileSystem ptr (#7180 )	4 years ago
wal_manager_test.cc	Use DbSessionId as cache key prefix when secondary cache is enabled (#8360 )	3 years ago
write_batch.cc	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033 )	4 years ago
write_batch_base.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
write_batch_internal.h	Integrity protection for live updates to WriteBatch (#7748 )	4 years ago
write_batch_test.cc	Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262 )	4 years ago
write_callback.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433 )	5 years ago
write_callback_test.cc	Move slow valgrind tests behind -DROCKSDB_FULL_VALGRIND_RUN (#8475 )	3 years ago
write_controller.cc	Revamp WriteController (#8064 )	4 years ago
write_controller.h	Revamp WriteController (#8064 )	4 years ago
write_controller_test.cc	Revamp WriteController (#8064 )	4 years ago
write_thread.cc	Stall writes in WriteBufferManager when memory_usage exceeds buffer_size (#7898 )	4 years ago
write_thread.h	typo: fix typo in db/write_thread's state (#8423 )	3 years ago