You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Baptiste Lemaire
e3a96c4823
Memtable sampling for mempurge heuristic. ( #8628 )
...
Summary:
Changes the API of the MemPurge process: the `bool experimental_allow_mempurge` and `experimental_mempurge_policy` flags have been replaced by a `double experimental_mempurge_threshold` option.
This change of API reflects another major change introduced in this PR: the MemPurgeDecider() function now works by sampling the memtables being flushed to estimate the overall amount of useful payload (payload minus the garbage), and then compare this useful payload estimate with the `double experimental_mempurge_threshold` value.
Therefore, when the value of this flag is `0.0` (default value), mempurge is simply deactivated. On the other hand, a value of `DBL_MAX` would be equivalent to always going through a mempurge regardless of the garbage ratio estimate.
At the moment, a `double experimental_mempurge_threshold` value else than 0.0 or `DBL_MAX` is opnly supported`with the `SkipList` memtable representation.
Regarding the sampling, this PR includes the introduction of a `MemTable::UniqueRandomSample` function that collects (approximately) random entries from the memtable by using the new `SkipList::Iterator::RandomSeek()` under the hood, or by iterating through each memtable entry, depending on the target sample size and the total number of entries.
The unit tests have been readapted to support this new API.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/8628
Reviewed By: pdillinger
Differential Revision: D30149315
Pulled By: bjlemaire
fbshipit-source-id: 1feef5390c95db6f4480ab4434716533d3947f27
3 years ago
..
advisor
remediation of S205607
4 years ago
block_cache_analyzer
Fix cmake build failure with gflags ( #8324 )
4 years ago
dump
Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE ( #6433 )
5 years ago
rdb
Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE ( #6433 )
5 years ago
CMakeLists.txt
Mark dependencies as PRIVATE and fix missing dependencies in tools. ( #6790 )
5 years ago
Dockerfile
adding docker build script and dockerfile
10 years ago
analyze_txn_stress_test.sh
Add copyright headers per FB open-source checkup tool. ( #5199 )
6 years ago
auto_sanity_test.sh
Add copyright headers per FB open-source checkup tool. ( #5199 )
6 years ago
backup_db.sh
Revamp check_format_compatible.sh ( #8012 )
4 years ago
benchmark.sh
Update benchmark.sh ( #8615 )
3 years ago
benchmark_leveldb.sh
Add copyright headers per FB open-source checkup tool. ( #5199 )
6 years ago
blob_dump.cc
Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE ( #6433 )
5 years ago
check_all_python.py
Allow missing "unversioned" python, as in CentOS 8 ( #6883 )
5 years ago
check_format_compatible.sh
Add 6.18, 6.19 and 6.20 to check_format_compatible.sh ( #8236 )
4 years ago
db_bench.cc
Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE ( #6433 )
5 years ago
db_bench_tool.cc
Memtable sampling for mempurge heuristic. ( #8628 )
3 years ago
db_bench_tool_test.cc
Add support for the integrated BlobDB to db_bench ( #7956 )
4 years ago
db_crashtest.py
Memtable sampling for mempurge heuristic. ( #8628 )
3 years ago
db_repl_stress.cc
Migrate away from Travis+Linux+amd64 ( #7791 )
4 years ago
db_sanity_test.cc
Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE ( #6433 )
5 years ago
dbench_monitor
Fix /bin/bash shebangs
7 years ago
generate_random_db.sh
Add copyright headers per FB open-source checkup tool. ( #5199 )
6 years ago
ingest_external_sst.sh
Add copyright headers per FB open-source checkup tool. ( #5199 )
6 years ago
io_tracer_parser.cc
Add IO Tracer Parser ( #7333 )
4 years ago
io_tracer_parser_test.cc
Fix cmake build failure with gflags ( #8324 )
4 years ago
io_tracer_parser_tool.cc
Add request_id in IODebugContext. ( #8045 )
4 years ago
io_tracer_parser_tool.h
Add IO Tracer Parser ( #7333 )
4 years ago
ldb.cc
Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE ( #6433 )
5 years ago
ldb_cmd.cc
Allow WAL dir to change with db dir ( #8582 )
3 years ago
ldb_cmd_impl.h
Some fixes and enhancements to `ldb repair` ( #8544 )
3 years ago
ldb_cmd_test.cc
Allow WAL dir to change with db dir ( #8582 )
3 years ago
ldb_test.py
Add list live files metadata ( #8446 )
3 years ago
ldb_tool.cc
Add list live files metadata ( #8446 )
3 years ago
pflag
Fix /bin/bash shebangs
7 years ago
reduce_levels_test.cc
Add more tests for assert status checked ( #7524 )
4 years ago
regression_test.sh
Update internal build script ( #7957 )
4 years ago
report_lite_binary_size.sh
Add copyright headers per FB open-source checkup tool. ( #5199 )
6 years ago
restore_db.sh
Revamp check_format_compatible.sh ( #8012 )
4 years ago
rocksdb_dump_test.sh
Add copyright headers per FB open-source checkup tool. ( #5199 )
6 years ago
run_flash_bench.sh
Add copyright headers per FB open-source checkup tool. ( #5199 )
6 years ago
run_leveldb.sh
Add copyright headers per FB open-source checkup tool. ( #5199 )
6 years ago
sample-dump.dmp
First version of rocksdb_dump and rocksdb_undump.
10 years ago
simulated_hybrid_file_system.cc
Write file temperature information to manifest ( #8284 )
4 years ago
simulated_hybrid_file_system.h
Write file temperature information to manifest ( #8284 )
4 years ago
sst_dump.cc
Implement a new subcommand "identify" for sst_dump ( #6943 )
5 years ago
sst_dump_test.cc
Add CreateFrom methods to Env/FileSystem ( #8174 )
3 years ago
sst_dump_tool.cc
Add CreateFrom methods to Env/FileSystem ( #8174 )
3 years ago
trace_analyzer.cc
Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE ( #6433 )
5 years ago
trace_analyzer_test.cc
Analyze MultiGet in trace_analyzer ( #8575 )
3 years ago
trace_analyzer_tool.cc
Analyze MultiGet in trace_analyzer ( #8575 )
3 years ago
trace_analyzer_tool.h
Analyze MultiGet in trace_analyzer ( #8575 )
3 years ago
verify_random_db.sh
Add copyright headers per FB open-source checkup tool. ( #5199 )
6 years ago
write_external_sst.sh
Revamp check_format_compatible.sh ( #8012 )
4 years ago
write_stress.cc
Add a SystemClock class to capture the time functions of an Env ( #7858 )
4 years ago
write_stress_runner.py
Allow missing "unversioned" python, as in CentOS 8 ( #6883 )
5 years ago