You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Hui Xiao
d665afdbf3
Account memory of FileMetaData in global memory limit (#9924)
Summary:
**Context/Summary:**
As revealed by heap profiling, allocation of `FileMetaData` for [newly created file added to a Version](https://github.com/facebook/rocksdb/pull/9924/files#diff-a6aa385940793f95a2c5b39cc670bd440c4547fa54fd44622f756382d5e47e43R774) can consume significant heap memory. This PR is to account that toward our global memory limit based on block cache capacity.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9924
Test Plan:
- Previous `make check` verified there are only 2 places where the memory of the allocated `FileMetaData` can be released
- New unit test `TEST_P(ChargeFileMetadataTestWithParam, Basic)`
- db bench (CPU cost of `charge_file_metadata` in write and compact)
- **write micros/op: -0.24%** : `TEST_TMPDIR=/dev/shm/testdb ./db_bench -benchmarks=fillseq -db=$TEST_TMPDIR -charge_file_metadata=1 (remove this option for pre-PR) -disable_auto_compactions=1 -write_buffer_size=100000 -num=4000000 | egrep 'fillseq'`
- **compact micros/op -0.87%** : `TEST_TMPDIR=/dev/shm/testdb ./db_bench -benchmarks=fillseq -db=$TEST_TMPDIR -charge_file_metadata=1 -disable_auto_compactions=1 -write_buffer_size=100000 -num=4000000 -numdistinct=1000 && ./db_bench -benchmarks=compact -db=$TEST_TMPDIR -use_existing_db=1 -charge_file_metadata=1 -disable_auto_compactions=1 | egrep 'compact'`
table 1 - write
#-run | (pre-PR) avg micros/op | std micros/op | (post-PR) micros/op | std micros/op | change (%)
-- | -- | -- | -- | -- | --
10 | 3.9711 | 0.264408 | 3.9914 | 0.254563 | 0.5111933721
20 | 3.83905 | 0.0664488 | 3.8251 | 0.0695456 | -0.3633711465
40 | 3.86625 | 0.136669 | 3.8867 | 0.143765 | 0.5289363078
80 | 3.87828 | 0.119007 | 3.86791 | 0.115674 | **-0.2673865734**
160 | 3.87677 | 0.162231 | 3.86739 | 0.16663 | **-0.2419539978**
table 2 - compact
#-run | (pre-PR) avg micros/op | std micros/op | (post-PR) micros/op | std micros/op | change (%)
-- | -- | -- | -- | -- | --
10 | 2,399,650.00 | 96,375.80 | 2,359,537.00 | 53,243.60 | -1.67
20 | 2,410,480.00 | 89,988.00 | 2,433,580.00 | 91,121.20 | 0.96
40 | 2.41E+06 | 121811 | 2.39E+06 | 131525 | **-0.96**
80 | 2.40E+06 | 134503 | 2.39E+06 | 108799 | **-0.78**
- stress test: `python3 tools/db_crashtest.py blackbox --charge_file_metadata=1 --cache_size=1` killed as normal
Reviewed By: ajkr
Differential Revision: D36055583
Pulled By: hx235
fbshipit-source-id: b60eab94707103cb1322cf815f05810ef0232625
|
2 years ago |
| .. |
|
CMakeLists.txt
|
Stress test for RocksDB transactions (#8936)
|
3 years ago |
|
batched_ops_stress.cc
|
Update test for secondary instance in stress test (#10121)
|
2 years ago |
|
cf_consistency_stress.cc
|
Update test for secondary instance in stress test (#10121)
|
2 years ago |
|
db_stress.cc
|
Add (& fix) some simple source code checks (#8821)
|
3 years ago |
|
db_stress_common.cc
|
Snapshots with user-specified timestamps (#9879)
|
2 years ago |
|
db_stress_common.h
|
Account memory of FileMetaData in global memory limit (#9924)
|
2 years ago |
|
db_stress_compaction_filter.h
|
Rename kRemoveWithSingleDelete to kPurge (#9951)
|
3 years ago |
|
db_stress_driver.cc
|
Snapshots with user-specified timestamps (#9879)
|
2 years ago |
|
db_stress_driver.h
|
Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)
|
5 years ago |
|
db_stress_env_wrapper.h
|
Make the Env class Customizable (#9293)
|
3 years ago |
|
db_stress_gflags.cc
|
Account memory of FileMetaData in global memory limit (#9924)
|
2 years ago |
|
db_stress_listener.cc
|
Adjust public APIs to prefer 128-bit SST unique ID (#10009)
|
3 years ago |
|
db_stress_listener.h
|
Avoid usage of ReopenWritableFile in db_stress (#9649)
|
3 years ago |
|
db_stress_shared_state.cc
|
Remove ROCKSDB_SUPPORT_THREAD_LOCAL define because it's a part of C++11 (#10015)
|
3 years ago |
|
db_stress_shared_state.h
|
Remove ROCKSDB_SUPPORT_THREAD_LOCAL define because it's a part of C++11 (#10015)
|
3 years ago |
|
db_stress_stat.cc
|
Fix Statistics in db_stress (#9260)
|
3 years ago |
|
db_stress_stat.h
|
Fix Statistics in db_stress (#9260)
|
3 years ago |
|
db_stress_table_properties_collector.h
|
Fix and detect headers with missing dependencies (#8893)
|
3 years ago |
|
db_stress_test_base.cc
|
Account memory of FileMetaData in global memory limit (#9924)
|
2 years ago |
|
db_stress_test_base.h
|
Fix a race condition in transaction stress test (#10157)
|
2 years ago |
|
db_stress_tool.cc
|
Snapshots with user-specified timestamps (#9879)
|
2 years ago |
|
expected_state.cc
|
Remove own ToString() (#9955)
|
3 years ago |
|
expected_state.h
|
db_stress verify with lost unsynced operations (#8966)
|
3 years ago |
|
multi_ops_txns_stress.cc
|
Fix a race condition in transaction stress test (#10157)
|
2 years ago |
|
multi_ops_txns_stress.h
|
Fix a bug of CompactionIterator/CompactionFilter using `Delete` (#9929)
|
3 years ago |
|
no_batched_ops_stress.cc
|
Account memory of FileMetaData in global memory limit (#9924)
|
2 years ago |
