You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Hui Xiao
d8c043f7ad
Trigger FIFO file deletion in non L0 only if exceeding max_table_files_size (#10955)
Summary:
**Context**
https://github.com/facebook/rocksdb/pull/10348 allows multi-level FIFO but accidentally made change to the logic of deleting files in `FIFOCompactionPicker::PickSizeCompaction`. With [this](https://github.com/facebook/rocksdb/pull/10348/files#diff-d8fb3d50749aa69b378de447e3d9cf2f48abe0281437f010b5d61365a7b813fdR156) and [this](https://github.com/facebook/rocksdb/pull/10348/files#diff-d8fb3d50749aa69b378de447e3d9cf2f48abe0281437f010b5d61365a7b813fdR235) together, it deletes one file in non-L0 even when `total_size <= mutable_cf_options.compaction_options_fifo.max_table_files_size`, which is incorrect.
As a consequence, FIFO exercises more file deletion in our crash testing, which is not able to verify correctly on deleted keys in the file deleted by compaction. This results in errors `error : inconsistent values for key 000000000000239F000000000000012B000000000000028B: expected state has the key, Get() returns NotFound.
Verification failed :(` or `Expected state has key 00000000000023A90000000000000003787878, iterator is at key 00000000000023A9000000000000004178
Column family: default, op_logs: S 00000000000023A90000000000000003787878`
**Summary**:
- Delete file for non-L0 only if `total_size <= mutable_cf_options.compaction_options_fifo.max_table_files_size`
- Add some helpful log to LOG file
Pull Request resolved: https://github.com/facebook/rocksdb/pull/10955
Test Plan:
- Errors repro-ed by
```
./db_stress --preserve_unverified_changes=1 --acquire_snapshot_one_in=10000 --adaptive_readahead=0 --allow_concurrent_memtable_write=0 --allow_data_in_errors=True --async_io=0 --avoid_flush_during_recovery=0 --avoid_unnecessary_blocking_io=1 --backup_max_size=104857600 --backup_one_in=100000 --batch_protection_bytes_per_key=0 --block_size=16384 --bloom_bits=10 --bottommost_compression_type=none --bytes_per_sync=0 --cache_index_and_filter_blocks=0 --cache_size=8388608 --cache_type=lru_cache --charge_compression_dictionary_building_buffer=1 --charge_file_metadata=1 --charge_filter_construction=0 --charge_table_reader=1 --checkpoint_one_in=1000000 --checksum_type=kxxHash --clear_column_family_one_in=0 --column_families=1 --compact_files_one_in=1000000 --compact_range_one_in=1000000 --compaction_pri=3 --compaction_style=2 --compaction_ttl=0 --compression_max_dict_buffer_bytes=8589934591 --compression_max_dict_bytes=16384 --compression_parallel_threads=1 --compression_type=xpress --compression_use_zstd_dict_trainer=1 --compression_zstd_max_train_bytes=0 --continuous_verification_interval=0 --data_block_index_type=0 --db=/dev/shm/rocksdb_test/rocksdb_crashtest_whitebox --db_write_buffer_size=1048576 --delpercent=0 --delrangepercent=0 --destroy_db_initially=1 --detect_filter_construct_corruption=0 --disable_wal=0 --enable_compaction_filter=0 --enable_pipelined_write=1 --expected_values_dir=/dev/shm/rocksdb_test/rocksdb_crashtest_expected --fail_if_options_file_error=1 --fifo_allow_compaction=1 --file_checksum_impl=xxh64 --flush_one_in=1000000 --format_version=4 --get_current_wal_file_one_in=0 --get_live_files_one_in=1000000 --get_property_one_in=1000000 --get_sorted_wal_files_one_in=0 --index_block_restart_interval=10 --index_type=2 --ingest_external_file_one_in=1000000 --initial_auto_readahead_size=16384 --iterpercent=10 --key_len_percent_dist=1,30,69 --level_compaction_dynamic_level_bytes=False --log2_keys_per_lock=10 --long_running_snapshots=0 --manual_wal_flush_one_in=0 --mark_for_compaction_one_file_in=10 --max_auto_readahead_size=524288 --max_background_compactions=1 --max_bytes_for_level_base=67108864 --max_key=25000000 --max_key_len=3 --max_manifest_file_size=1073741824 --max_write_batch_group_size_bytes=1048576 --max_write_buffer_number=3 --max_write_buffer_size_to_maintain=0 --memtable_prefix_bloom_size_ratio=0.01 --memtable_protection_bytes_per_key=1 --memtable_whole_key_filtering=1 --memtablerep=skip_list --min_write_buffer_number_to_merge=2 --mmap_read=0 --mock_direct_io=True --nooverwritepercent=0 --num_file_reads_for_auto_readahead=2 --open_files=-1 --open_metadata_write_fault_one_in=0 --open_read_fault_one_in=0 --open_write_fault_one_in=0 --ops_per_thread=40000 --optimize_filters_for_memory=0 --paranoid_file_checks=1 --partition_filters=0 --partition_pinning=3 --pause_background_one_in=1000000 --periodic_compaction_seconds=0 --prefix_size=7 --prefixpercent=5 --prepopulate_block_cache=0 --preserve_internal_time_seconds=3600 --progress_reports=0 --read_fault_one_in=1000 --readahead_size=0 --readpercent=65 --recycle_log_file_num=1 --reopen=0 --ribbon_starting_level=999 --secondary_cache_fault_one_in=0 --snapshot_hold_ops=100000 --sst_file_manager_bytes_per_sec=0 --sst_file_manager_bytes_per_truncate=0 --stats_dump_period_sec=0 --subcompactions=2 --sync=0 --sync_fault_injection=0 --target_file_size_base=16777216 --target_file_size_multiplier=1 --test_batches_snapshots=0 --top_level_index_pinning=1 --unpartitioned_pinning=1 --use_direct_io_for_flush_and_compaction=1 --use_direct_reads=1 --use_full_merge_v1=1 --use_merge=0 --use_multiget=0 --use_put_entity_one_in=0 --user_timestamp_size=0 --value_size_mult=32 --verify_checksum=1 --verify_checksum_one_in=1000000 --verify_db_one_in=100000 --verify_iterator_with_expected_state_one_in=0 --verify_sst_unique_id_in_manifest=1 --wal_bytes_per_sync=0 --wal_compression=none --write_buffer_size=33554432 --write_dbid_to_manifest=1 --writepercent=20
```
is gone after this fix
- CI
Reviewed By: ajkr
Differential Revision: D41319441
Pulled By: hx235
fbshipit-source-id: 6939753767007f7449ea7055b1420aabd03d7709
|
2 years ago |
| .. |
|
clipping_iterator.h
|
Make InternalKeyComparator not configurable (#10342)
|
2 years ago |
|
clipping_iterator_test.cc
|
Print stack traces on frozen tests in CI (#10828)
|
2 years ago |
|
compaction.cc
|
Support tiering when file endpoints overlap (#10961)
|
2 years ago |
|
compaction.h
|
Allow penultimate level output for the last level only compaction (#10822)
|
2 years ago |
|
compaction_iteration_stats.h
|
Support readahead during compaction for blob files (#9187)
|
3 years ago |
|
compaction_iterator.cc
|
Fix CompactionIterator flag for penultimate level output (#10967)
|
2 years ago |
|
compaction_iterator.h
|
Add option `preserve_internal_time_seconds` to preserve the time info (#10747)
|
2 years ago |
|
compaction_iterator_test.cc
|
Basic Support for Merge with user-defined timestamp (#10819)
|
2 years ago |
|
compaction_job.cc
|
clang-format for db/compaction (#10882)
|
2 years ago |
|
compaction_job.h
|
clang-format for db/compaction (#10882)
|
2 years ago |
|
compaction_job_stats_test.cc
|
clang-format for db/compaction (#10882)
|
2 years ago |
|
compaction_job_test.cc
|
clang-format for db/compaction (#10882)
|
2 years ago |
|
compaction_outputs.cc
|
Use `sstableKeyCompare()` for compaction output boundary check (#10763)
|
2 years ago |
|
compaction_outputs.h
|
Use `sstableKeyCompare()` for compaction output boundary check (#10763)
|
2 years ago |
|
compaction_picker.cc
|
Fix FIFO causing overlapping seqnos in L0 files due to overlapped seqnos between ingested files and memtable's (#10777)
|
2 years ago |
|
compaction_picker.h
|
Fix FIFO causing overlapping seqnos in L0 files due to overlapped seqnos between ingested files and memtable's (#10777)
|
2 years ago |
|
compaction_picker_fifo.cc
|
Trigger FIFO file deletion in non L0 only if exceeding max_table_files_size (#10955)
|
2 years ago |
|
compaction_picker_fifo.h
|
Fix FIFO causing overlapping seqnos in L0 files due to overlapped seqnos between ingested files and memtable's (#10777)
|
2 years ago |
|
compaction_picker_level.cc
|
Fix FIFO causing overlapping seqnos in L0 files due to overlapped seqnos between ingested files and memtable's (#10777)
|
2 years ago |
|
compaction_picker_level.h
|
Fix FIFO causing overlapping seqnos in L0 files due to overlapped seqnos between ingested files and memtable's (#10777)
|
2 years ago |
|
compaction_picker_test.cc
|
Fix FIFO causing overlapping seqnos in L0 files due to overlapped seqnos between ingested files and memtable's (#10777)
|
2 years ago |
|
compaction_picker_universal.cc
|
clang-format for db/compaction (#10882)
|
2 years ago |
|
compaction_picker_universal.h
|
Fix FIFO causing overlapping seqnos in L0 files due to overlapped seqnos between ingested files and memtable's (#10777)
|
2 years ago |
|
compaction_service_job.cc
|
Improve SubCompaction Partitioning (#10393)
|
2 years ago |
|
compaction_service_test.cc
|
Deflake CompactionServiceTest.BasicCompactions (#10697)
|
2 years ago |
|
compaction_state.cc
|
Tiered Compaction: per key placement support (#9964)
|
2 years ago |
|
compaction_state.h
|
Tiered Compaction: per key placement support (#9964)
|
2 years ago |
|
file_pri.h
|
Try to start TTL earlier with kMinOverlappingRatio is used (#8749)
|
3 years ago |
|
sst_partitioner.cc
|
Restore Regex support for ObjectLibrary::Register, rename new APIs to allow old one to be deprecated in the future (#9362)
|
3 years ago |
|
subcompaction_state.cc
|
Refactor Compaction file cut `ShouldStopBefore()` (#10629)
|
2 years ago |
|
subcompaction_state.h
|
Set correct temperature for range tombstone only file in penultimate level (#10972)
|
2 years ago |
|
tiered_compaction_test.cc
|
Support tiering when file endpoints overlap (#10961)
|
2 years ago |
