You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Changyu Bi
333abe9c55
Ignore max_compaction_bytes for compaction input that are within output key-range (#10835)
Summary:
When picking compaction input files, we sometimes stop picking a file that is fully included in the output key-range due to hitting max_compaction_bytes. Including these input files can potentially reduce WA at the expense of larger compactions. Larger compaction should be fine as files from input level are usually 10X smaller than files from output level. This PR adds a mutable CF option `ignore_max_compaction_bytes_for_input` that is enabled by default. We can remove this option once we are sure it is safe.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/10835
Test Plan:
- CI, a unit test on max_compaction_bytes fails before turning this flag off.
- Benchmark does not show much difference in WA: `./db_bench --benchmarks=fillrandom,waitforcompaction,stats,levelstats -max_background_jobs=12 -num=2000000000 -target_file_size_base=33554432 --write_buffer_size=33554432`
```
main:
** Compaction Stats [default] **
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
L0 3/0 91.59 MB 0.8 70.9 0.0 70.9 200.8 129.9 0.0 1.5 25.2 71.2 2886.55 2463.45 9725 0.297 1093M 254K 0.0 0.0
L1 9/0 248.03 MB 1.0 392.0 129.8 262.2 391.7 129.5 0.0 3.0 69.0 68.9 5821.71 5536.90 804 7.241 6029M 5814K 0.0 0.0
L2 87/0 2.50 GB 1.0 537.0 128.5 408.5 533.8 125.2 0.7 4.2 69.5 69.1 7912.24 7323.70 4417 1.791 8299M 36M 0.0 0.0
L3 836/0 24.99 GB 1.0 616.9 118.3 498.7 594.5 95.8 5.2 5.0 66.9 64.5 9442.38 8490.28 4204 2.246 9749M 306M 0.0 0.0
L4 2355/0 62.95 GB 0.3 67.3 37.1 30.2 54.2 24.0 38.9 1.5 72.2 58.2 954.37 821.18 917 1.041 1076M 173M 0.0 0.0
Sum 3290/0 90.77 GB 0.0 1684.2 413.7 1270.5 1775.0 504.5 44.9 13.7 63.8 67.3 27017.25 24635.52 20067 1.346 26G 522M 0.0 0.0
Cumulative compaction: 1774.96 GB write, 154.29 MB/s write, 1684.19 GB read, 146.40 MB/s read, 27017.3 seconds
This PR:
** Compaction Stats [default] **
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
L0 3/0 45.71 MB 0.8 72.9 0.0 72.9 202.8 129.9 0.0 1.6 25.4 70.7 2938.16 2510.36 9741 0.302 1124M 265K 0.0 0.0
L1 8/0 234.54 MB 0.9 384.5 129.8 254.7 384.2 129.6 0.0 3.0 69.0 68.9 5708.08 5424.43 791 7.216 5913M 5753K 0.0 0.0
L2 84/0 2.47 GB 1.0 543.1 128.6 414.5 539.9 125.4 0.7 4.2 69.6 69.2 7989.31 7403.13 4418 1.808 8393M 36M 0.0 0.0
L3 839/0 24.96 GB 1.0 615.6 118.4 497.2 593.2 96.0 5.1 5.0 66.6 64.1 9471.23 8489.31 4193 2.259 9726M 306M 0.0 0.0
L4 2360/0 63.04 GB 0.3 67.6 37.3 30.3 54.4 24.1 38.9 1.5 71.5 57.6 967.30 827.99 907 1.066 1080M 173M 0.0 0.0
Sum 3294/0 90.75 GB 0.0 1683.8 414.2 1269.6 1774.5 504.9 44.8 13.7 63.7 67.1 27074.08 24655.22 20050 1.350 26G 522M 0.0 0.0
Cumulative compaction: 1774.52 GB write, 157.09 MB/s write, 1683.77 GB read, 149.06 MB/s read, 27074.1 seconds
```
Reviewed By: ajkr
Differential Revision: D40518319
Pulled By: cbi42
fbshipit-source-id: f4ea614bc0ebefe007ffaf05bb9aec9a8ca25b60
|
2 years ago |
| .. |
|
clipping_iterator.h
|
Make InternalKeyComparator not configurable (#10342)
|
2 years ago |
|
clipping_iterator_test.cc
|
Print stack traces on frozen tests in CI (#10828)
|
2 years ago |
|
compaction.cc
|
Allow the last level data moving up to penultimate level (#10782)
|
2 years ago |
|
compaction.h
|
Exclude timestamp when checking compaction boundaries (#10787)
|
2 years ago |
|
compaction_iteration_stats.h
|
Support readahead during compaction for blob files (#9187)
|
3 years ago |
|
compaction_iterator.cc
|
Fix no internal time recorded for small preclude_last_level (#10829)
|
2 years ago |
|
compaction_iterator.h
|
Add option `preserve_internal_time_seconds` to preserve the time info (#10747)
|
2 years ago |
|
compaction_iterator_test.cc
|
Print stack traces on frozen tests in CI (#10828)
|
2 years ago |
|
compaction_job.cc
|
Add option `preserve_internal_time_seconds` to preserve the time info (#10747)
|
2 years ago |
|
compaction_job.h
|
Add option `preserve_internal_time_seconds` to preserve the time info (#10747)
|
2 years ago |
|
compaction_job_stats_test.cc
|
Remove own ToString() (#9955)
|
3 years ago |
|
compaction_job_test.cc
|
Print stack traces on frozen tests in CI (#10828)
|
2 years ago |
|
compaction_outputs.cc
|
Use `sstableKeyCompare()` for compaction output boundary check (#10763)
|
2 years ago |
|
compaction_outputs.h
|
Use `sstableKeyCompare()` for compaction output boundary check (#10763)
|
2 years ago |
|
compaction_picker.cc
|
Ignore max_compaction_bytes for compaction input that are within output key-range (#10835)
|
2 years ago |
|
compaction_picker.h
|
Support subcmpct using reserved resources for round-robin priority (#10341)
|
2 years ago |
|
compaction_picker_fifo.cc
|
Enable a multi-level db to smoothly migrate to FIFO via DB::Open (#10348)
|
2 years ago |
|
compaction_picker_fifo.h
|
Add OpenAndTrimHistory API to support trimming data with specified timestamp (#9410)
|
3 years ago |
|
compaction_picker_level.cc
|
RoundRobin TTL compaction (#10725)
|
2 years ago |
|
compaction_picker_level.h
|
Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262)
|
4 years ago |
|
compaction_picker_test.cc
|
Ignore max_compaction_bytes for compaction input that are within output key-range (#10835)
|
2 years ago |
|
compaction_picker_universal.cc
|
Fix overlapping check by excluding timestamp (#10615)
|
2 years ago |
|
compaction_picker_universal.h
|
Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262)
|
4 years ago |
|
compaction_service_job.cc
|
Improve SubCompaction Partitioning (#10393)
|
2 years ago |
|
compaction_service_test.cc
|
Deflake CompactionServiceTest.BasicCompactions (#10697)
|
2 years ago |
|
compaction_state.cc
|
Tiered Compaction: per key placement support (#9964)
|
2 years ago |
|
compaction_state.h
|
Tiered Compaction: per key placement support (#9964)
|
2 years ago |
|
file_pri.h
|
Try to start TTL earlier with kMinOverlappingRatio is used (#8749)
|
3 years ago |
|
sst_partitioner.cc
|
Restore Regex support for ObjectLibrary::Register, rename new APIs to allow old one to be deprecated in the future (#9362)
|
3 years ago |
|
subcompaction_state.cc
|
Refactor Compaction file cut `ShouldStopBefore()` (#10629)
|
2 years ago |
|
subcompaction_state.h
|
Refactor Compaction file cut `ShouldStopBefore()` (#10629)
|
2 years ago |
|
tiered_compaction_test.cc
|
Fix no internal time recorded for small preclude_last_level (#10829)
|
2 years ago |
