fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Levi Tamasi 68d8b28389 Log the amount of blob garbage generated by compactions in the MANIFEST (#8450) ... Summary: The patch builds on `BlobGarbageMeter` and `BlobCountingIterator` (introduced in https://github.com/facebook/rocksdb/issues/8426 and https://github.com/facebook/rocksdb/issues/8443 respectively) and ties it all together. It measures the amount of garbage generated by a compaction and logs the corresponding `BlobFileGarbage` records as part of the compaction job's `VersionEdit`. Note: in order to have accurate results, `kRemoveAndSkipUntil` for compaction filters is implemented using iteration. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8450 Test Plan: Ran `make check` and the crash test script. Reviewed By: jay-zhuang Differential Revision: D29338207 Pulled By: ltamasi fbshipit-source-id: 4381c432ac215139439f6d6fb801a6c0e4d8c128		3 years ago
..
blob	Log the amount of blob garbage generated by compactions in the MANIFEST (#8450)	3 years ago
compaction	Log the amount of blob garbage generated by compactions in the MANIFEST (#8450)	3 years ago
db_impl	Fix DeleteFilesInRange may cause inconsistent compaction error (#8434)	3 years ago
arena_wrapped_db_iter.cc	Rename ImmutableOptions variables (#8409)	3 years ago
arena_wrapped_db_iter.h	Rename ImmutableOptions variables (#8409)	3 years ago
builder.cc	Added memtable garbage statistics (#8411)	3 years ago
builder.h	Added memtable garbage statistics (#8411)	3 years ago
c.cc	Expose CompressionOptions::parallel_threads through C API (#8302)	4 years ago
c_test.c	Expose CompressionOptions::parallel_threads through C API (#8302)	4 years ago
column_family.cc	Use DbSessionId as cache key prefix when secondary cache is enabled (#8360)	3 years ago
column_family.h	Use DbSessionId as cache key prefix when secondary cache is enabled (#8360)	3 years ago
column_family_test.cc	Add CreateFrom methods to Env/FileSystem (#8174)	3 years ago
compact_files_test.cc	Compaction should not move data to up level (#8116)	4 years ago
comparator_db_test.cc	Remove unused includes (#7604)	4 years ago
convenience.cc	Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262)	4 years ago
corruption_test.cc	Add CreateFrom methods to Env/FileSystem (#8174)	3 years ago
cuckoo_table_db_test.cc	Revert "Turn on memtable bloom filter by default. (#6584)" (#7939)	4 years ago
db_basic_test.cc	Initial support for secondary cache in LRUCache (#8271)	4 years ago
db_block_cache_test.cc	Cache warming data blocks during flush (#8242)	3 years ago
db_bloom_filter_test.cc	Add table properties for number of entries added to filters (#8323)	4 years ago
db_compaction_filter_test.cc	Allow applying `CompactionFilter` outside of compaction (#8243)	4 years ago
db_compaction_test.cc	Fix DeleteFilesInRange may cause inconsistent compaction error (#8434)	3 years ago
db_dynamic_level_test.cc	Add a SystemClock class to capture the time functions of an Env (#7858)	4 years ago
db_encryption_test.cc	Improvements to Env::GetChildren (#7819)	4 years ago
db_filesnapshot.cc	Fix `GetLiveFiles()` returning OPTIONS-000000 (#8268)	4 years ago
db_flush_test.cc	Added memtable garbage statistics (#8411)	3 years ago
db_info_dumper.cc	Fix write-ahead log file size overflow (#7870)	4 years ago
db_info_dumper.h	Add a DB Session ID (#6959)	4 years ago
db_inplace_update_test.cc	Whole DBTest to skip fsync (#7274)	4 years ago
db_io_failure_test.cc	Add more tests to ASSERT_STATUS_CHECKED (3), API change (#7715)	4 years ago
db_iter.cc	Rename ImmutableOptions variables (#8409)	3 years ago
db_iter.h	Rename ImmutableOptions variables (#8409)	3 years ago
db_iter_stress_test.cc	Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262)	4 years ago
db_iter_test.cc	Rename ImmutableOptions variables (#8409)	3 years ago
db_iterator_test.cc	Instantiate tests DBIteratorTestForPinnedData (#8051)	4 years ago
db_kv_checksum_test.cc	Integrity protection for live updates to WriteBatch (#7748)	4 years ago
db_log_iter_test.cc	Add a SystemClock class to capture the time functions of an Env (#7858)	4 years ago
db_logical_block_size_cache_test.cc	Add further tests to ASSERT_STATUS_CHECKED (2) (#7698)	4 years ago
db_memtable_test.cc	Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262)	4 years ago
db_merge_operand_test.cc	Support for Merge in Integrated BlobDB with base values (#8292)	3 years ago
db_merge_operator_test.cc	Add further tests to ASSERT_STATUS_CHECKED (1) (#7679)	4 years ago
db_options_test.cc	Add ObjectRegistry to ConfigOptions (#8166)	4 years ago
db_properties_test.cc	Assert unlimited max_open_files for FIFO compaction. (#8172)	4 years ago
db_range_del_test.cc	Fix manual compaction `max_compaction_bytes` under-calculated issue (#8269)	4 years ago
db_secondary_test.cc	Add remote compaction public API (#8300)	4 years ago
db_sst_test.cc	Fix flaky failure in DBSSTest.DBWithSstFileManagerForBlobFilesWithGC (#8196)	4 years ago
db_statistics_test.cc	Add more tests for assert status checked (#7524)	4 years ago
db_table_properties_test.cc	Add more tests for assert status checked (#7524)	4 years ago
db_tailing_iter_test.cc	Add more tests to ASSERT_STATUS_CHECKED (3), API change (#7715)	4 years ago
db_test.cc	Print blob file checksums as hex (#8437)	3 years ago
db_test2.cc	Cancel compact range (#8351)	3 years ago
db_test_util.cc	Add CreateFrom methods to Env/FileSystem (#8174)	3 years ago
db_test_util.h	Cache warming data blocks during flush (#8242)	3 years ago
db_universal_compaction_test.cc	Revert "Turn on memtable bloom filter by default. (#6584)" (#7939)	4 years ago
db_wal_test.cc	Use DbSessionId as cache key prefix when secondary cache is enabled (#8360)	3 years ago
db_with_timestamp_basic_test.cc	Fix a bug in key comparison when index type is kBinarySearchWithFirstKey (#8062)	4 years ago
db_with_timestamp_compaction_test.cc	Whole DBTest to skip fsync (#7274)	4 years ago
db_write_buffer_manager_test.cc	Stall writes in WriteBufferManager when memory_usage exceeds buffer_size (#7898)	4 years ago
db_write_test.cc	Add more tests to ASSERT_STATUS_CHECKED (4) (#7718)	4 years ago
dbformat.cc	Make CompactRange and GetApproximateSizes work with timestamp (#7684)	4 years ago
dbformat.h	Fix some typos in comments (#8066)	4 years ago
dbformat_test.cc	Remove unused includes (#7604)	4 years ago
deletefile_test.cc	Add more tests to ASSERT_STATUS_CHECKED (4) (#7718)	4 years ago
error_handler.cc	All the NoSpace() errors will be handled by regular SetBGError and RecoverFromNoSpace() (#8376)	3 years ago
error_handler.h	Fix some typos in comments (#8066)	4 years ago
error_handler_fs_test.cc	All the NoSpace() errors will be handled by regular SetBGError and RecoverFromNoSpace() (#8376)	3 years ago
event_helpers.cc	Add table properties for number of entries added to filters (#8323)	4 years ago
event_helpers.h	Pass SST file checksum information through OnTableFileCreated (#7108)	4 years ago
experimental.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
external_sst_file_basic_test.cc	Sync ingested files only if reopen is supported by the FS (#8296)	4 years ago
external_sst_file_ingestion_job.cc	Sync ingested files only if reopen is supported by the FS (#8296)	4 years ago
external_sst_file_ingestion_job.h	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033)	4 years ago
external_sst_file_test.cc	Deflake ExternalSSTFileTest.PickedLevelBug (#8307)	4 years ago
fault_injection_test.cc	Add more tests for assert status checked (#7524)	4 years ago
file_indexer.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
file_indexer.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
file_indexer_test.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
filename_test.cc	Remove unused includes (#7604)	4 years ago
flush_job.cc	Added memtable garbage statistics (#8411)	3 years ago
flush_job.h	Use SST file manager to track blob files as well (#8037)	4 years ago
flush_job_test.cc	Use DbSessionId as cache key prefix when secondary cache is enabled (#8360)	3 years ago
flush_scheduler.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
flush_scheduler.h	Include C++ standard library headers instead of C compatibility headers (#8068)	4 years ago
forward_iterator.cc	Fix some typos in comments (#8066)	4 years ago
forward_iterator.h	Properly report IO errors when IndexType::kBinarySearchWithFirstKey is used (#6621)	5 years ago
forward_iterator_bench.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
import_column_family_job.cc	Add a SystemClock class to capture the time functions of an Env (#7858)	4 years ago
import_column_family_job.h	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033)	4 years ago
import_column_family_test.cc	No elide constructors (#7798)	4 years ago
internal_stats.cc	Pin CacheEntryStatsCollector to fix performance bug (#8385)	3 years ago
internal_stats.h	Pin CacheEntryStatsCollector to fix performance bug (#8385)	3 years ago
job_context.h	Rename ImmutableOptions variables (#8409)	3 years ago
kv_checksum.h	Integrity protection for live updates to WriteBatch (#7748)	4 years ago
listener_test.cc	Add more tests for assert status checked (#7524)	4 years ago
log_format.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
log_reader.cc	Fix kPointInTimeRecovery handling of truncated WAL (#7701)	4 years ago
log_reader.h	Real fix for race in backup custom checksum checking (#7309)	4 years ago
log_test.cc	Make StringEnv, StringSink, StringSource use FS classes (#7786)	4 years ago
log_writer.cc	No elide constructors (#7798)	4 years ago
log_writer.h	Include C++ standard library headers instead of C compatibility headers (#8068)	4 years ago
logs_with_prep_tracker.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
logs_with_prep_tracker.h	Include C++ standard library headers instead of C compatibility headers (#8068)	4 years ago
lookup_key.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
malloc_stats.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
malloc_stats.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
manual_compaction_test.cc	Add more tests for assert status checked (#7524)	4 years ago
memtable.cc	Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262)	4 years ago
memtable.h	Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262)	4 years ago
memtable_list.cc	Fix some typos in comments (#8066)	4 years ago
memtable_list.h	Add initial blob support to batched MultiGet (#7766)	4 years ago
memtable_list_test.cc	Use DbSessionId as cache key prefix when secondary cache is enabled (#8360)	3 years ago
merge_context.h	Add Merge Operator support to WriteBatchWithIndex (#8135)	4 years ago
merge_helper.cc	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033)	4 years ago
merge_helper.h	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033)	4 years ago
merge_helper_test.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
merge_operator.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
merge_test.cc	MergeHelper::FilterMerge() calling ElapsedNanosSafe() upon exit even … (#7867)	4 years ago
obsolete_files_test.cc	Print blob file checksums as hex (#8437)	3 years ago
options_file_test.cc	No elide constructors (#7798)	4 years ago
output_validator.cc	Use NPHash64 in more places (#7632)	4 years ago
output_validator.h	Add remote compaction public API (#8300)	4 years ago
perf_context_test.cc	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033)	4 years ago
periodic_work_scheduler.cc	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033)	4 years ago
periodic_work_scheduler.h	Add a SystemClock class to capture the time functions of an Env (#7858)	4 years ago
periodic_work_scheduler_test.cc	Add a SystemClock class to capture the time functions of an Env (#7858)	4 years ago
pinned_iterators_manager.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
plain_table_db_test.cc	Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262)	4 years ago
pre_release_callback.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
prefix_test.cc	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033)	4 years ago
range_del_aggregator.cc	In ParseInternalKey(), include corrupt key info in Status (#7515)	4 years ago
range_del_aggregator.h	Fix some typos in comments (#8066)	4 years ago
range_del_aggregator_bench.cc	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033)	4 years ago
range_del_aggregator_test.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
range_tombstone_fragmenter.cc	Added memtable garbage statistics (#8411)	3 years ago
range_tombstone_fragmenter.h	Added memtable garbage statistics (#8411)	3 years ago
range_tombstone_fragmenter_test.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
read_callback.h	Get() with timestamp should respect snapshot (#7227)	4 years ago
repair.cc	Use DbSessionId as cache key prefix when secondary cache is enabled (#8360)	3 years ago
repair_test.cc	Use SST file manager to track blob files as well (#8037)	4 years ago
snapshot_checker.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
snapshot_impl.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
snapshot_impl.h	Fix some typos in comments (#8066)	4 years ago
table_cache.cc	Use DbSessionId as cache key prefix when secondary cache is enabled (#8360)	3 years ago
table_cache.h	Use DbSessionId as cache key prefix when secondary cache is enabled (#8360)	3 years ago
table_properties_collector.cc	Apply `sample_for_compression` to all block-based tables (#8105)	4 years ago
table_properties_collector.h	Make it possible to apply only a subrange of table property collectors (#8298)	4 years ago
table_properties_collector_test.cc	Make it possible to apply only a subrange of table property collectors (#8298)	4 years ago
transaction_log_impl.cc	Add more tests for assert status checked (#7524)	4 years ago
transaction_log_impl.h	Store FileSystemPtr object that contains FileSystem ptr (#7180)	4 years ago
trim_history_scheduler.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
trim_history_scheduler.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
version_builder.cc	Handle blob files when options.best_efforts_recovery is true (#8180)	4 years ago
version_builder.h	Handle blob files when options.best_efforts_recovery is true (#8180)	4 years ago
version_builder_test.cc	Print blob file checksums as hex (#8437)	3 years ago
version_edit.cc	Write file temperature information to manifest (#8284)	4 years ago
version_edit.h	Write file temperature information to manifest (#8284)	4 years ago
version_edit_handler.cc	Fixed manifest_dump issues when printing keys and values containing null characters (#8378)	3 years ago
version_edit_handler.h	Fixed manifest_dump issues when printing keys and values containing null characters (#8378)	3 years ago
version_edit_test.cc	Make it able to ignore WAL related VersionEdits in older versions (#7873)	4 years ago
version_set.cc	Remove obsolete method VersionSet::VerifyCompactionFileConsistency (#8449)	3 years ago
version_set.h	Remove obsolete method VersionSet::VerifyCompactionFileConsistency (#8449)	3 years ago
version_set_test.cc	Print blob file checksums as hex (#8437)	3 years ago
wal_edit.cc	Always track WAL obsoletion (#7759)	4 years ago
wal_edit.h	Always track WAL obsoletion (#7759)	4 years ago
wal_edit_test.cc	Always track WAL obsoletion (#7759)	4 years ago
wal_manager.cc	Make types of Immutable/Mutable Options fields match that of the underlying Option (#8176)	4 years ago
wal_manager.h	Store FileSystemPtr object that contains FileSystem ptr (#7180)	4 years ago
wal_manager_test.cc	Use DbSessionId as cache key prefix when secondary cache is enabled (#8360)	3 years ago
write_batch.cc	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033)	4 years ago
write_batch_base.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
write_batch_internal.h	Integrity protection for live updates to WriteBatch (#7748)	4 years ago
write_batch_test.cc	Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262)	4 years ago
write_callback.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
write_callback_test.cc	Add more tests for assert status checked (#7524)	4 years ago
write_controller.cc	Revamp WriteController (#8064)	4 years ago
write_controller.h	Revamp WriteController (#8064)	4 years ago
write_controller_test.cc	Revamp WriteController (#8064)	4 years ago
write_thread.cc	Stall writes in WriteBufferManager when memory_usage exceeds buffer_size (#7898)	4 years ago
write_thread.h	typo: fix typo in db/write_thread's state (#8423)	3 years ago