You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Hui Xiao
a5063c8931
Fix issue of opening too many files in BlockBasedTableReaderCapMemoryTest.CapMemoryUsageUnderCacheCapacity (#9869)
Summary:
**Context:**
Unit test for https://github.com/facebook/rocksdb/pull/9748 keeps opening new files to see whether the new feature is able to correctly constrain the opening based on block cache capacity.
However, the unit test has two places written inefficiently that can lead to opening too many new files relative to underlying operating system/file system constraint, even before hitting the block cache capacity:
(1) [opened_table_reader_num < 2 * max_table_reader_num](https://github.com/facebook/rocksdb/pull/9748/files?show-viewed-files=true&file-filters%5B%5D=#diff-ec9f5353e317df71093094734ba29193b94a998f0f9c9af924e4c99692195eeaR438), which can leads to 1200 + open files because of (2) below
(2) NewLRUCache(6 * CacheReservationManagerImpl<CacheEntryRole::kBlockBasedTableReader>::GetDummyEntrySize()) in [here](https://github.com/facebook/rocksdb/pull/9748/files?show-viewed-files=true&file-filters%5B%5D=#diff-ec9f5353e317df71093094734ba29193b94a998f0f9c9af924e4c99692195eeaR364)
Therefore we see CI failures like this on machine with a strict open file limit ~1000 (see the "table_1021" naming in following error msg)
https://app.circleci.com/pipelines/github/facebook/rocksdb/12886/workflows/75524682-3fa4-41ee-9a61-81827b51d99b/jobs/345270
```
fs_->NewWritableFile(path, foptions, &file, nullptr)
IO error: While open a file for appending: /dev/shm/rocksdb.Jedwt/run-block_based_table_reader_test-CapMemoryUsageUnderCacheCapacity-BlockBasedTableReaderCapMemoryTest.CapMemoryUsageUnderCacheCapacity-0/block_based_table_reader_test_1668910_829492452552920927/**table_1021**: Too many open files
```
**Summary:**
- Revised the test more efficiently on the above 2 places, including using 1.1 instead 2 in the threshold and lowering down the block cache capacity a bit
- Renamed some variables for clarity
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9869
Test Plan:
- Manual inspection of max opened table reader in all test case, which is around ~389
- Circle CI to see if error is gone
Reviewed By: ajkr
Differential Revision: D35752655
Pulled By: hx235
fbshipit-source-id: 8a0953d39d561babfa4257b8ed8550bb21b04839
|
3 years ago |
| .. |
|
adaptive
|
More refactoring ahead of footer & meta changes (#9240)
|
3 years ago |
|
block_based
|
Fix issue of opening too many files in BlockBasedTableReaderCapMemoryTest.CapMemoryUsageUnderCacheCapacity (#9869)
|
3 years ago |
|
cuckoo
|
Add rate limiter priority to ReadOptions (#9424)
|
3 years ago |
|
plain
|
Meta-internal folly integration with F14FastMap (#9546)
|
3 years ago |
|
block_fetcher.cc
|
Fix segfault in FilePrefetchBuffer with async_io enabled (#9777)
|
3 years ago |
|
block_fetcher.h
|
More refactoring ahead of footer & meta changes (#9240)
|
3 years ago |
|
block_fetcher_test.cc
|
Make MemoryAllocator into a Customizable class (#8980)
|
3 years ago |
|
cleanable_test.cc
|
Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)
|
5 years ago |
|
format.cc
|
Add rate limiter priority to ReadOptions (#9424)
|
3 years ago |
|
format.h
|
Optimize & clean up footer code (#9280)
|
3 years ago |
|
get_context.cc
|
Support readahead during compaction for blob files (#9187)
|
3 years ago |
|
get_context.h
|
Cleanup includes in dbformat.h (#8930)
|
3 years ago |
|
internal_iterator.h
|
Reuse internal auto readhead_size at each Level (expect L0) for Iterations (#9056)
|
3 years ago |
|
iter_heap.h
|
Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)
|
5 years ago |
|
iterator.cc
|
Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)
|
5 years ago |
|
iterator_wrapper.h
|
Reuse internal auto readhead_size at each Level (expect L0) for Iterations (#9056)
|
3 years ago |
|
merger_test.cc
|
Cleanup multiple implementations of VectorIterator (#8901)
|
3 years ago |
|
merging_iterator.cc
|
MergingIterator: rearrange fields to reduce paddings (#9024)
|
3 years ago |
|
merging_iterator.h
|
Cleanup includes in dbformat.h (#8930)
|
3 years ago |
|
meta_blocks.cc
|
Tests for filter compatibility (#9773)
|
3 years ago |
|
meta_blocks.h
|
Tests for filter compatibility (#9773)
|
3 years ago |
|
mock_table.cc
|
Add rate limiter priority to ReadOptions (#9424)
|
3 years ago |
|
mock_table.h
|
Fix some minor issues in the Customizable infrastructure (#8566)
|
3 years ago |
|
multiget_context.h
|
Fix major bug with MultiGet, DeleteRange, and memtable Bloom (#9453)
|
3 years ago |
|
persistent_cache_helper.cc
|
New stable, fixed-length cache keys (#9126)
|
3 years ago |
|
persistent_cache_helper.h
|
Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)
|
5 years ago |
|
persistent_cache_options.h
|
New stable, fixed-length cache keys (#9126)
|
3 years ago |
|
scoped_arena_iterator.h
|
Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)
|
5 years ago |
|
sst_file_dumper.cc
|
New backup meta schema, with file temperatures (#9660)
|
3 years ago |
|
sst_file_dumper.h
|
New backup meta schema, with file temperatures (#9660)
|
3 years ago |
|
sst_file_reader.cc
|
Fast path for detecting unchanged prefix_extractor (#9407)
|
3 years ago |
|
sst_file_reader_test.cc
|
Use the comparator from the sst file table properties in sst_dump_tool (#9491)
|
3 years ago |
|
sst_file_writer.cc
|
`compression_per_level` should be used for flush and changeable (#9658)
|
3 years ago |
|
sst_file_writer_collectors.h
|
Cleanup includes in dbformat.h (#8930)
|
3 years ago |
|
table_builder.h
|
Fast path for detecting unchanged prefix_extractor (#9407)
|
3 years ago |
|
table_factory.cc
|
Restore Regex support for ObjectLibrary::Register, rename new APIs to allow old one to be deprecated in the future (#9362)
|
3 years ago |
|
table_properties.cc
|
Fix a bug of TEST_SetRandomTableProperties due to non-zero padding between fields in TableProperties struct (#9812)
|
3 years ago |
|
table_properties_internal.h
|
Improve / clean up meta block code & integrity (#9163)
|
3 years ago |
|
table_reader.h
|
dedup ReadOptions in iterator hierarchy (#7210)
|
4 years ago |
|
table_reader_bench.cc
|
Fast path for detecting unchanged prefix_extractor (#9407)
|
3 years ago |
|
table_reader_caller.h
|
Fix and detect headers with missing dependencies (#8893)
|
3 years ago |
|
table_test.cc
|
Remove BlockBasedTableOptions.hash_index_allow_collision (#9454)
|
3 years ago |
|
two_level_iterator.cc
|
Clarify caching behavior for index and filter partitions (#9068)
|
3 years ago |
|
two_level_iterator.h
|
Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)
|
5 years ago |
|
unique_id.cc
|
Experimental support for SST unique IDs (#8990)
|
3 years ago |
|
unique_id_impl.h
|
Experimental support for SST unique IDs (#8990)
|
3 years ago |
