|
|
|
|
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
|
|
|
|
|
// This source code is licensed under both the GPLv2 (found in the
|
|
|
|
|
// COPYING file in the root directory) and Apache 2.0 License
|
|
|
|
|
// (found in the LICENSE.Apache file in the root directory).
|
|
|
|
|
//
|
|
|
|
|
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
|
|
|
|
|
// Use of this source code is governed by a BSD-style license that can be
|
|
|
|
|
// found in the LICENSE file. See the AUTHORS file for names of contributors.
|
|
|
|
|
|
|
|
|
|
#include "rocksdb/options.h"
|
|
|
|
|
|
|
|
|
|
#include <cinttypes>
|
|
|
|
|
#include <limits>
|
|
|
|
|
|
|
|
|
|
#include "logging/logging.h"
|
|
|
|
|
#include "monitoring/statistics.h"
|
|
|
|
|
#include "options/db_options.h"
|
|
|
|
|
#include "options/options_helper.h"
|
|
|
|
|
#include "rocksdb/cache.h"
|
|
|
|
|
#include "rocksdb/compaction_filter.h"
|
|
|
|
|
#include "rocksdb/comparator.h"
|
|
|
|
|
#include "rocksdb/env.h"
|
|
|
|
|
#include "rocksdb/filter_policy.h"
|
|
|
|
|
#include "rocksdb/memtablerep.h"
|
|
|
|
|
#include "rocksdb/merge_operator.h"
|
|
|
|
|
#include "rocksdb/slice.h"
|
|
|
|
|
#include "rocksdb/slice_transform.h"
|
|
|
|
|
#include "rocksdb/sst_file_manager.h"
|
|
|
|
|
#include "rocksdb/sst_partitioner.h"
|
|
|
|
|
#include "rocksdb/table.h"
|
|
|
|
|
#include "rocksdb/table_properties.h"
|
|
|
|
|
#include "rocksdb/wal_filter.h"
|
|
|
|
|
#include "table/block_based/block_based_table_factory.h"
|
|
|
|
|
#include "util/compression.h"
|
|
|
|
|
|
|
|
|
|
namespace ROCKSDB_NAMESPACE {
|
|
|
|
|
|
|
|
|
|
AdvancedColumnFamilyOptions::AdvancedColumnFamilyOptions() {
|
[RocksDB] [Column Family] Interface proposal
Summary:
<This diff is for Column Family branch>
Sharing some of the work I've done so far. This diff compiles and passes the tests.
The biggest change is in options.h - I broke down Options into two parts - DBOptions and ColumnFamilyOptions. DBOptions is DB-specific (env, create_if_missing, block_cache, etc.) and ColumnFamilyOptions is column family-specific (all compaction options, compresion options, etc.). Note that this does not break backwards compatibility at all.
Further, I created DBWithColumnFamily which inherits DB interface and adds new functions with column family support. Clients can transparently switch to DBWithColumnFamily and it will not break their backwards compatibility.
There are few methods worth checking out: ListColumnFamilies(), MultiNewIterator(), MultiGet() and GetSnapshot(). [GetSnapshot() returns the snapshot across all column families for now - I think that's what we agreed on]
Finally, I made small changes to WriteBatch so we are able to atomically insert data across column families.
Please provide feedback.
Test Plan: make check works, the code is backward compatible
Reviewers: dhruba, haobo, sdong, kailiu, emayanke
CC: leveldb
Differential Revision: https://reviews.facebook.net/D14445
11 years ago
|
|
|
assert(memtable_factory.get() != nullptr);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
AdvancedColumnFamilyOptions::AdvancedColumnFamilyOptions(const Options& options)
|
|
|
|
|
: max_write_buffer_number(options.max_write_buffer_number),
|
|
|
|
|
min_write_buffer_number_to_merge(
|
|
|
|
|
options.min_write_buffer_number_to_merge),
|
Support saving history in memtable_list
Summary:
For transactions, we are using the memtables to validate that there are no write conflicts. But after flushing, we don't have any memtables, and transactions could fail to commit. So we want to someone keep around some extra history to use for conflict checking. In addition, we want to provide a way to increase the size of this history if too many transactions fail to commit.
After chatting with people, it seems like everyone prefers just using Memtables to store this history (instead of a separate history structure). It seems like the best place for this is abstracted inside the memtable_list. I decide to create a separate list in MemtableListVersion as using the same list complicated the flush/installalflushresults logic too much.
This diff adds a new parameter to control how much memtable history to keep around after flushing. However, it sounds like people aren't too fond of adding new parameters. So I am making the default size of flushed+not-flushed memtables be set to max_write_buffers. This should not change the maximum amount of memory used, but make it more likely we're using closer the the limit. (We are now postponing deleting flushed memtables until the max_write_buffer limit is reached). So while we might use more memory on average, we are still obeying the limit set (and you could argue it's better to go ahead and use up memory now instead of waiting for a write stall to happen to test this limit).
However, if people are opposed to this default behavior, we can easily set it to 0 and require this parameter be set in order to use transactions.
Test Plan: Added a xfunc test to play around with setting different values of this parameter in all tests. Added testing in memtablelist_test and planning on adding more testing here.
Reviewers: sdong, rven, igor
Reviewed By: igor
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D37443
10 years ago
|
|
|
max_write_buffer_number_to_maintain(
|
|
|
|
|
options.max_write_buffer_number_to_maintain),
|
Refactor trimming logic for immutable memtables (#5022)
Summary:
MyRocks currently sets `max_write_buffer_number_to_maintain` in order to maintain enough history for transaction conflict checking. The effectiveness of this approach depends on the size of memtables. When memtables are small, it may not keep enough history; when memtables are large, this may consume too much memory.
We are proposing a new way to configure memtable list history: by limiting the memory usage of immutable memtables. The new option is `max_write_buffer_size_to_maintain` and it will take precedence over the old `max_write_buffer_number_to_maintain` if they are both set to non-zero values. The new option accounts for the total memory usage of flushed immutable memtables and mutable memtable. When the total usage exceeds the limit, RocksDB may start dropping immutable memtables (which is also called trimming history), starting from the oldest one.
The semantics of the old option actually works both as an upper bound and lower bound. History trimming will start if number of immutable memtables exceeds the limit, but it will never go below (limit-1) due to history trimming.
In order the mimic the behavior with the new option, history trimming will stop if dropping the next immutable memtable causes the total memory usage go below the size limit. For example, assuming the size limit is set to 64MB, and there are 3 immutable memtables with sizes of 20, 30, 30. Although the total memory usage is 80MB > 64MB, dropping the oldest memtable will reduce the memory usage to 60MB < 64MB, so in this case no memtable will be dropped.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5022
Differential Revision: D14394062
Pulled By: miasantreble
fbshipit-source-id: 60457a509c6af89d0993f988c9b5c2aa9e45f5c5
5 years ago
|
|
|
max_write_buffer_size_to_maintain(
|
|
|
|
|
options.max_write_buffer_size_to_maintain),
|
|
|
|
|
inplace_update_support(options.inplace_update_support),
|
|
|
|
|
inplace_update_num_locks(options.inplace_update_num_locks),
|
|
|
|
|
inplace_callback(options.inplace_callback),
|
|
|
|
|
memtable_prefix_bloom_size_ratio(
|
|
|
|
|
options.memtable_prefix_bloom_size_ratio),
|
|
|
|
|
memtable_whole_key_filtering(options.memtable_whole_key_filtering),
|
|
|
|
|
memtable_huge_page_size(options.memtable_huge_page_size),
|
|
|
|
|
memtable_insert_with_hint_prefix_extractor(
|
|
|
|
|
options.memtable_insert_with_hint_prefix_extractor),
|
|
|
|
|
bloom_locality(options.bloom_locality),
|
|
|
|
|
arena_block_size(options.arena_block_size),
|
|
|
|
|
compression_per_level(options.compression_per_level),
|
|
|
|
|
num_levels(options.num_levels),
|
|
|
|
|
level0_slowdown_writes_trigger(options.level0_slowdown_writes_trigger),
|
|
|
|
|
level0_stop_writes_trigger(options.level0_stop_writes_trigger),
|
|
|
|
|
target_file_size_base(options.target_file_size_base),
|
|
|
|
|
target_file_size_multiplier(options.target_file_size_multiplier),
|
options.level_compaction_dynamic_level_bytes to allow RocksDB to pick size bases of levels dynamically.
Summary:
When having fixed max_bytes_for_level_base, the ratio of size of largest level and the second one can range from 0 to the multiplier. This makes LSM tree frequently irregular and unpredictable. It can also cause poor space amplification in some cases.
In this improvement (proposed by Igor Kabiljo), we introduce a parameter option.level_compaction_use_dynamic_max_bytes. When turning it on, RocksDB is free to pick a level base in the range of (options.max_bytes_for_level_base/options.max_bytes_for_level_multiplier, options.max_bytes_for_level_base] so that real level ratios are close to options.max_bytes_for_level_multiplier.
Test Plan: New unit tests and pass tests suites including valgrind.
Reviewers: MarkCallaghan, rven, yhchiang, igor, ikabiljo
Reviewed By: ikabiljo
Subscribers: yoshinorim, ikabiljo, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D31437
10 years ago
|
|
|
level_compaction_dynamic_level_bytes(
|
|
|
|
|
options.level_compaction_dynamic_level_bytes),
|
|
|
|
|
max_bytes_for_level_multiplier(options.max_bytes_for_level_multiplier),
|
|
|
|
|
max_bytes_for_level_multiplier_additional(
|
|
|
|
|
options.max_bytes_for_level_multiplier_additional),
|
|
|
|
|
max_compaction_bytes(options.max_compaction_bytes),
|
|
|
|
|
soft_pending_compaction_bytes_limit(
|
|
|
|
|
options.soft_pending_compaction_bytes_limit),
|
|
|
|
|
hard_pending_compaction_bytes_limit(
|
|
|
|
|
options.hard_pending_compaction_bytes_limit),
|
|
|
|
|
compaction_style(options.compaction_style),
|
|
|
|
|
compaction_pri(options.compaction_pri),
|
|
|
|
|
compaction_options_universal(options.compaction_options_universal),
|
|
|
|
|
compaction_options_fifo(options.compaction_options_fifo),
|
|
|
|
|
max_sequential_skip_in_iterations(
|
|
|
|
|
options.max_sequential_skip_in_iterations),
|
|
|
|
|
memtable_factory(options.memtable_factory),
|
TablePropertiesCollectorFactory
Summary:
This diff addresses task #4296714 and rethinks how users provide us with TablePropertiesCollectors as part of Options.
Here's description of task #4296714:
I'm debugging #4295529 and noticed that our count of user properties kDeletedKeys is wrong. We're sharing one single InternalKeyPropertiesCollector with all Table Builders. In LOG Files, we're outputting number of kDeletedKeys as connected with a single table, while it's actually the total count of deleted keys since creation of the DB.
For example, this table has 3155 entries and 1391828 deleted keys.
The problem with current approach that we call methods on a single TablePropertiesCollector for all the tables we create. Even worse, we could do it from multiple threads at the same time and TablePropertiesCollector has no way of knowing which table we're calling it for.
Good part: Looks like nobody inside Facebook is using Options::table_properties_collectors. This means we should be able to painfully change the API.
In this change, I introduce TablePropertiesCollectorFactory. For every table we create, we call `CreateTablePropertiesCollector`, which creates a TablePropertiesCollector for a single table. We then use it sequentially from a single thread, which means it doesn't have to be thread-safe.
Test Plan:
Added a test in table_properties_collector_test that fails on master (build two tables, assert that kDeletedKeys count is correct for the second one).
Also, all other tests
Reviewers: sdong, dhruba, haobo, kailiu
Reviewed By: kailiu
CC: leveldb
Differential Revision: https://reviews.facebook.net/D18579
11 years ago
|
|
|
table_properties_collector_factories(
|
|
|
|
|
options.table_properties_collector_factories),
|
|
|
|
|
max_successive_merges(options.max_successive_merges),
|
|
|
|
|
optimize_filters_for_hits(options.optimize_filters_for_hits),
|
Add options.compaction_measure_io_stats to print write I/O stats in compactions
Summary:
Add options.compaction_measure_io_stats to print out / pass to listener accumulated time spent on write calls. Example outputs in info logs:
2015/08/12-16:27:59.463944 7fd428bff700 (Original Log Time 2015/08/12-16:27:59.463922) EVENT_LOG_v1 {"time_micros": 1439422079463897, "job": 6, "event": "compaction_finished", "output_level": 1, "num_output_files": 4, "total_output_size": 6900525, "num_input_records": 111483, "num_output_records": 106877, "file_write_nanos": 15663206, "file_range_sync_nanos": 649588, "file_fsync_nanos": 349614797, "file_prepare_write_nanos": 1505812, "lsm_state": [2, 4, 0, 0, 0, 0, 0]}
Add two more counters in iostats_context.
Also add a parameter of db_bench.
Test Plan: Add a unit test. Also manually verify LOG outputs in db_bench
Subscribers: leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D44115
9 years ago
|
|
|
paranoid_file_checks(options.paranoid_file_checks),
|
|
|
|
|
force_consistency_checks(options.force_consistency_checks),
|
|
|
|
|
report_bg_io_stats(options.report_bg_io_stats),
|
|
|
|
|
ttl(options.ttl),
|
Periodic Compactions (#5166)
Summary:
Introducing Periodic Compactions.
This feature allows all the files in a CF to be periodically compacted. It could help in catching any corruptions that could creep into the DB proactively as every file is constantly getting re-compacted. And also, of course, it helps to cleanup data older than certain threshold.
- Introduced a new option `periodic_compaction_time` to control how long a file can live without being compacted in a CF.
- This works across all levels.
- The files are put in the same level after going through the compaction. (Related files in the same level are picked up as `ExpandInputstoCleanCut` is used).
- Compaction filters, if any, are invoked as usual.
- A new table property, `file_creation_time`, is introduced to implement this feature. This property is set to the time at which the SST file was created (and that time is given by the underlying Env/OS).
This feature can be enabled on its own, or in conjunction with `ttl`. It is possible to set a different time threshold for the bottom level when used in conjunction with ttl. Since `ttl` works only on 0 to last but one levels, you could set `ttl` to, say, 1 day, and `periodic_compaction_time` to, say, 7 days. Since `ttl < periodic_compaction_time` all files in last but one levels keep getting picked up based on ttl, and almost never based on periodic_compaction_time. The files in the bottom level get picked up for compaction based on `periodic_compaction_time`.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5166
Differential Revision: D14884441
Pulled By: sagar0
fbshipit-source-id: 408426cbacb409c06386a98632dcf90bfa1bda47
6 years ago
|
|
|
periodic_compaction_seconds(options.periodic_compaction_seconds),
|
|
|
|
|
sample_for_compression(options.sample_for_compression),
|
|
|
|
|
enable_blob_files(options.enable_blob_files),
|
|
|
|
|
min_blob_size(options.min_blob_size),
|
|
|
|
|
blob_file_size(options.blob_file_size),
|
|
|
|
|
blob_compression_type(options.blob_compression_type),
|
|
|
|
|
enable_blob_garbage_collection(options.enable_blob_garbage_collection),
|
|
|
|
|
blob_garbage_collection_age_cutoff(
|
Make it possible to force the garbage collection of the oldest blob files (#8994)
Summary:
The current BlobDB garbage collection logic works by relocating the valid
blobs from the oldest blob files as they are encountered during compaction,
and cleaning up blob files once they contain nothing but garbage. However,
with sufficiently skewed workloads, it is theoretically possible to end up in a
situation when few or no compactions get scheduled for the SST files that contain
references to the oldest blob files, which can lead to increased space amp due
to the lack of GC.
In order to efficiently handle such workloads, the patch adds a new BlobDB
configuration option called `blob_garbage_collection_force_threshold`,
which signals to BlobDB to schedule targeted compactions for the SST files
that keep alive the oldest batch of blob files if the overall ratio of garbage in
the given blob files meets the threshold *and* all the given blob files are
eligible for GC based on `blob_garbage_collection_age_cutoff`. (For example,
if the new option is set to 0.9, targeted compactions will get scheduled if the
sum of garbage bytes meets or exceeds 90% of the sum of total bytes in the
oldest blob files, assuming all affected blob files are below the age-based cutoff.)
The net result of these targeted compactions is that the valid blobs in the oldest
blob files are relocated and the oldest blob files themselves cleaned up (since
*all* SST files that rely on them get compacted away).
These targeted compactions are similar to periodic compactions in the sense
that they force certain SST files that otherwise would not get picked up to undergo
compaction and also in the sense that instead of merging files from multiple levels,
they target a single file. (Note: such compactions might still include neighboring files
from the same level due to the need of having a "clean cut" boundary but they never
include any files from any other level.)
This functionality is currently only supported with the leveled compaction style
and is inactive by default (since the default value is set to 1.0, i.e. 100%).
Pull Request resolved: https://github.com/facebook/rocksdb/pull/8994
Test Plan: Ran `make check` and tested using `db_bench` and the stress/crash tests.
Reviewed By: riversand963
Differential Revision: D31489850
Pulled By: ltamasi
fbshipit-source-id: 44057d511726a0e2a03c5d9313d7511b3f0c4eab
3 years ago
|
|
|
options.blob_garbage_collection_age_cutoff),
|
|
|
|
|
blob_garbage_collection_force_threshold(
|
|
|
|
|
options.blob_garbage_collection_force_threshold),
|
|
|
|
|
blob_compaction_readahead_size(options.blob_compaction_readahead_size) {
|
|
|
|
|
assert(memtable_factory.get() != nullptr);
|
|
|
|
|
if (max_bytes_for_level_multiplier_additional.size() <
|
|
|
|
|
static_cast<unsigned int>(num_levels)) {
|
|
|
|
|
max_bytes_for_level_multiplier_additional.resize(num_levels, 1);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
ColumnFamilyOptions::ColumnFamilyOptions()
|
|
|
|
|
: compression(Snappy_Supported() ? kSnappyCompression : kNoCompression),
|
|
|
|
|
table_factory(
|
|
|
|
|
std::shared_ptr<TableFactory>(new BlockBasedTableFactory())) {}
|
|
|
|
|
|
|
|
|
|
ColumnFamilyOptions::ColumnFamilyOptions(const Options& options)
|
|
|
|
|
: ColumnFamilyOptions(*static_cast<const ColumnFamilyOptions*>(&options)) {}
|
|
|
|
|
|
|
|
|
|
DBOptions::DBOptions() {}
|
|
|
|
|
DBOptions::DBOptions(const Options& options)
|
|
|
|
|
: DBOptions(*static_cast<const DBOptions*>(&options)) {}
|
|
|
|
|
|
|
|
|
|
void DBOptions::Dump(Logger* log) const {
|
|
|
|
|
ImmutableDBOptions(*this).Dump(log);
|
|
|
|
|
MutableDBOptions(*this).Dump(log);
|
|
|
|
|
} // DBOptions::Dump
|
|
|
|
|
|
|
|
|
|
void ColumnFamilyOptions::Dump(Logger* log) const {
|
|
|
|
|
ROCKS_LOG_HEADER(log, " Options.comparator: %s",
|
|
|
|
|
comparator->Name());
|
|
|
|
|
ROCKS_LOG_HEADER(log, " Options.merge_operator: %s",
|
|
|
|
|
merge_operator ? merge_operator->Name() : "None");
|
|
|
|
|
ROCKS_LOG_HEADER(log, " Options.compaction_filter: %s",
|
|
|
|
|
compaction_filter ? compaction_filter->Name() : "None");
|
|
|
|
|
ROCKS_LOG_HEADER(
|
|
|
|
|
log, " Options.compaction_filter_factory: %s",
|
|
|
|
|
compaction_filter_factory ? compaction_filter_factory->Name() : "None");
|
|
|
|
|
ROCKS_LOG_HEADER(
|
|
|
|
|
log, " Options.sst_partitioner_factory: %s",
|
|
|
|
|
sst_partitioner_factory ? sst_partitioner_factory->Name() : "None");
|
|
|
|
|
ROCKS_LOG_HEADER(log, " Options.memtable_factory: %s",
|
|
|
|
|
memtable_factory->Name());
|
|
|
|
|
ROCKS_LOG_HEADER(log, " Options.table_factory: %s",
|
|
|
|
|
table_factory->Name());
|
|
|
|
|
ROCKS_LOG_HEADER(log, " table_factory options: %s",
|
|
|
|
|
table_factory->GetPrintableOptions().c_str());
|
|
|
|
|
ROCKS_LOG_HEADER(log, " Options.write_buffer_size: %" ROCKSDB_PRIszt,
|
|
|
|
|
write_buffer_size);
|
|
|
|
|
ROCKS_LOG_HEADER(log, " Options.max_write_buffer_number: %d",
|
|
|
|
|
max_write_buffer_number);
|
|
|
|
|
if (!compression_per_level.empty()) {
|
|
|
|
|
for (unsigned int i = 0; i < compression_per_level.size(); i++) {
|
|
|
|
|
ROCKS_LOG_HEADER(
|
|
|
|
|
log, " Options.compression[%d]: %s", i,
|
|
|
|
|
CompressionTypeToString(compression_per_level[i]).c_str());
|
|
|
|
|
}
|
|
|
|
|
} else {
|
|
|
|
|
ROCKS_LOG_HEADER(log, " Options.compression: %s",
|
|
|
|
|
CompressionTypeToString(compression).c_str());
|
|
|
|
|
}
|
|
|
|
|
ROCKS_LOG_HEADER(
|
|
|
|
|
log, " Options.bottommost_compression: %s",
|
|
|
|
|
bottommost_compression == kDisableCompressionOption
|
|
|
|
|
? "Disabled"
|
|
|
|
|
: CompressionTypeToString(bottommost_compression).c_str());
|
|
|
|
|
ROCKS_LOG_HEADER(
|
|
|
|
|
log, " Options.prefix_extractor: %s",
|
|
|
|
|
prefix_extractor == nullptr ? "nullptr" : prefix_extractor->Name());
|
|
|
|
|
ROCKS_LOG_HEADER(log,
|
|
|
|
|
" Options.memtable_insert_with_hint_prefix_extractor: %s",
|
|
|
|
|
memtable_insert_with_hint_prefix_extractor == nullptr
|
|
|
|
|
? "nullptr"
|
|
|
|
|
: memtable_insert_with_hint_prefix_extractor->Name());
|
|
|
|
|
ROCKS_LOG_HEADER(log, " Options.num_levels: %d", num_levels);
|
|
|
|
|
ROCKS_LOG_HEADER(log, " Options.min_write_buffer_number_to_merge: %d",
|
|
|
|
|
min_write_buffer_number_to_merge);
|
|
|
|
|
ROCKS_LOG_HEADER(log, " Options.max_write_buffer_number_to_maintain: %d",
|
|
|
|
|
max_write_buffer_number_to_maintain);
|
Refactor trimming logic for immutable memtables (#5022)
Summary:
MyRocks currently sets `max_write_buffer_number_to_maintain` in order to maintain enough history for transaction conflict checking. The effectiveness of this approach depends on the size of memtables. When memtables are small, it may not keep enough history; when memtables are large, this may consume too much memory.
We are proposing a new way to configure memtable list history: by limiting the memory usage of immutable memtables. The new option is `max_write_buffer_size_to_maintain` and it will take precedence over the old `max_write_buffer_number_to_maintain` if they are both set to non-zero values. The new option accounts for the total memory usage of flushed immutable memtables and mutable memtable. When the total usage exceeds the limit, RocksDB may start dropping immutable memtables (which is also called trimming history), starting from the oldest one.
The semantics of the old option actually works both as an upper bound and lower bound. History trimming will start if number of immutable memtables exceeds the limit, but it will never go below (limit-1) due to history trimming.
In order the mimic the behavior with the new option, history trimming will stop if dropping the next immutable memtable causes the total memory usage go below the size limit. For example, assuming the size limit is set to 64MB, and there are 3 immutable memtables with sizes of 20, 30, 30. Although the total memory usage is 80MB > 64MB, dropping the oldest memtable will reduce the memory usage to 60MB < 64MB, so in this case no memtable will be dropped.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5022
Differential Revision: D14394062
Pulled By: miasantreble
fbshipit-source-id: 60457a509c6af89d0993f988c9b5c2aa9e45f5c5
5 years ago
|
|
|
ROCKS_LOG_HEADER(log,
|
|
|
|
|
" Options.max_write_buffer_size_to_maintain: %" PRIu64,
|
|
|
|
|
max_write_buffer_size_to_maintain);
|
|
|
|
|
ROCKS_LOG_HEADER(
|
|
|
|
|
log, " Options.bottommost_compression_opts.window_bits: %d",
|
|
|
|
|
bottommost_compression_opts.window_bits);
|
|
|
|
|
ROCKS_LOG_HEADER(
|
|
|
|
|
log, " Options.bottommost_compression_opts.level: %d",
|
|
|
|
|
bottommost_compression_opts.level);
|
|
|
|
|
ROCKS_LOG_HEADER(
|
|
|
|
|
log, " Options.bottommost_compression_opts.strategy: %d",
|
|
|
|
|
bottommost_compression_opts.strategy);
|
|
|
|
|
ROCKS_LOG_HEADER(
|
|
|
|
|
log,
|
|
|
|
|
" Options.bottommost_compression_opts.max_dict_bytes: "
|
|
|
|
|
"%" PRIu32,
|
|
|
|
|
bottommost_compression_opts.max_dict_bytes);
|
|
|
|
|
ROCKS_LOG_HEADER(
|
|
|
|
|
log,
|
|
|
|
|
" Options.bottommost_compression_opts.zstd_max_train_bytes: "
|
|
|
|
|
"%" PRIu32,
|
|
|
|
|
bottommost_compression_opts.zstd_max_train_bytes);
|
|
|
|
|
ROCKS_LOG_HEADER(
|
|
|
|
|
log,
|
|
|
|
|
" Options.bottommost_compression_opts.parallel_threads: "
|
|
|
|
|
"%" PRIu32,
|
|
|
|
|
bottommost_compression_opts.parallel_threads);
|
|
|
|
|
ROCKS_LOG_HEADER(
|
|
|
|
|
log, " Options.bottommost_compression_opts.enabled: %s",
|
|
|
|
|
bottommost_compression_opts.enabled ? "true" : "false");
|
Limit buffering for collecting samples for compression dictionary (#7970)
Summary:
For dictionary compression, we need to collect some representative samples of the data to be compressed, which we use to either generate or train (when `CompressionOptions::zstd_max_train_bytes > 0`) a dictionary. Previously, the strategy was to buffer all the data blocks during flush, and up to the target file size during compaction. That strategy allowed us to randomly pick samples from as wide a range as possible that'd be guaranteed to land in a single output file.
However, some users try to make huge files in memory-constrained environments, where this strategy can cause OOM. This PR introduces an option, `CompressionOptions::max_dict_buffer_bytes`, that limits how much data blocks are buffered before we switch to unbuffered mode (which means creating the per-SST dictionary, writing out the buffered data, and compressing/writing new blocks as soon as they are built). It is not strict as we currently buffer more than just data blocks -- also keys are buffered. But it does make a step towards giving users predictable memory usage.
Related changes include:
- Changed sampling for dictionary compression to select unique data blocks when there is limited availability of data blocks
- Made use of `BlockBuilder::SwapAndReset()` to save an allocation+memcpy when buffering data blocks for building a dictionary
- Changed `ParseBoolean()` to accept an input containing characters after the boolean. This is necessary since, with this PR, a value for `CompressionOptions::enabled` is no longer necessarily the final component in the `CompressionOptions` string.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7970
Test Plan:
- updated `CompressionOptions` unit tests to verify limit is respected (to the extent expected in the current implementation) in various scenarios of flush/compaction to bottommost/non-bottommost level
- looked at jemalloc heap profiles right before and after switching to unbuffered mode during flush/compaction. Verified memory usage in buffering is proportional to the limit set.
Reviewed By: pdillinger
Differential Revision: D26467994
Pulled By: ajkr
fbshipit-source-id: 3da4ef9fba59974e4ef40e40c01611002c861465
4 years ago
|
|
|
ROCKS_LOG_HEADER(
|
|
|
|
|
log,
|
|
|
|
|
" Options.bottommost_compression_opts.max_dict_buffer_bytes: "
|
|
|
|
|
"%" PRIu64,
|
|
|
|
|
bottommost_compression_opts.max_dict_buffer_bytes);
|
|
|
|
|
ROCKS_LOG_HEADER(log, " Options.compression_opts.window_bits: %d",
|
|
|
|
|
compression_opts.window_bits);
|
|
|
|
|
ROCKS_LOG_HEADER(log, " Options.compression_opts.level: %d",
|
|
|
|
|
compression_opts.level);
|
|
|
|
|
ROCKS_LOG_HEADER(log, " Options.compression_opts.strategy: %d",
|
|
|
|
|
compression_opts.strategy);
|
|
|
|
|
ROCKS_LOG_HEADER(
|
|
|
|
|
log,
|
|
|
|
|
" Options.compression_opts.max_dict_bytes: %" PRIu32,
|
Shared dictionary compression using reference block
Summary:
This adds a new metablock containing a shared dictionary that is used
to compress all data blocks in the SST file. The size of the shared dictionary
is configurable in CompressionOptions and defaults to 0. It's currently only
used for zlib/lz4/lz4hc, but the block will be stored in the SST regardless of
the compression type if the user chooses a nonzero dictionary size.
During compaction, computes the dictionary by randomly sampling the first
output file in each subcompaction. It pre-computes the intervals to sample
by assuming the output file will have the maximum allowable length. In case
the file is smaller, some of the pre-computed sampling intervals can be beyond
end-of-file, in which case we skip over those samples and the dictionary will
be a bit smaller. After the dictionary is generated using the first file in a
subcompaction, it is loaded into the compression library before writing each
block in each subsequent file of that subcompaction.
On the read path, gets the dictionary from the metablock, if it exists. Then,
loads that dictionary into the compression library before reading each block.
Test Plan: new unit test
Reviewers: yhchiang, IslamAbdelRahman, cyan, sdong
Reviewed By: sdong
Subscribers: andrewkr, yoshinorim, kradhakrishnan, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D52287
9 years ago
|
|
|
compression_opts.max_dict_bytes);
|
|
|
|
|
ROCKS_LOG_HEADER(log,
|
|
|
|
|
" Options.compression_opts.zstd_max_train_bytes: "
|
|
|
|
|
"%" PRIu32,
|
|
|
|
|
compression_opts.zstd_max_train_bytes);
|
|
|
|
|
ROCKS_LOG_HEADER(log,
|
|
|
|
|
" Options.compression_opts.parallel_threads: "
|
|
|
|
|
"%" PRIu32,
|
|
|
|
|
compression_opts.parallel_threads);
|
|
|
|
|
ROCKS_LOG_HEADER(log,
|
|
|
|
|
" Options.compression_opts.enabled: %s",
|
|
|
|
|
compression_opts.enabled ? "true" : "false");
|
Limit buffering for collecting samples for compression dictionary (#7970)
Summary:
For dictionary compression, we need to collect some representative samples of the data to be compressed, which we use to either generate or train (when `CompressionOptions::zstd_max_train_bytes > 0`) a dictionary. Previously, the strategy was to buffer all the data blocks during flush, and up to the target file size during compaction. That strategy allowed us to randomly pick samples from as wide a range as possible that'd be guaranteed to land in a single output file.
However, some users try to make huge files in memory-constrained environments, where this strategy can cause OOM. This PR introduces an option, `CompressionOptions::max_dict_buffer_bytes`, that limits how much data blocks are buffered before we switch to unbuffered mode (which means creating the per-SST dictionary, writing out the buffered data, and compressing/writing new blocks as soon as they are built). It is not strict as we currently buffer more than just data blocks -- also keys are buffered. But it does make a step towards giving users predictable memory usage.
Related changes include:
- Changed sampling for dictionary compression to select unique data blocks when there is limited availability of data blocks
- Made use of `BlockBuilder::SwapAndReset()` to save an allocation+memcpy when buffering data blocks for building a dictionary
- Changed `ParseBoolean()` to accept an input containing characters after the boolean. This is necessary since, with this PR, a value for `CompressionOptions::enabled` is no longer necessarily the final component in the `CompressionOptions` string.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7970
Test Plan:
- updated `CompressionOptions` unit tests to verify limit is respected (to the extent expected in the current implementation) in various scenarios of flush/compaction to bottommost/non-bottommost level
- looked at jemalloc heap profiles right before and after switching to unbuffered mode during flush/compaction. Verified memory usage in buffering is proportional to the limit set.
Reviewed By: pdillinger
Differential Revision: D26467994
Pulled By: ajkr
fbshipit-source-id: 3da4ef9fba59974e4ef40e40c01611002c861465
4 years ago
|
|
|
ROCKS_LOG_HEADER(log,
|
|
|
|
|
" Options.compression_opts.max_dict_buffer_bytes: "
|
|
|
|
|
"%" PRIu64,
|
|
|
|
|
compression_opts.max_dict_buffer_bytes);
|
|
|
|
|
ROCKS_LOG_HEADER(log, " Options.level0_file_num_compaction_trigger: %d",
|
|
|
|
|
level0_file_num_compaction_trigger);
|
|
|
|
|
ROCKS_LOG_HEADER(log, " Options.level0_slowdown_writes_trigger: %d",
|
|
|
|
|
level0_slowdown_writes_trigger);
|
|
|
|
|
ROCKS_LOG_HEADER(log, " Options.level0_stop_writes_trigger: %d",
|
|
|
|
|
level0_stop_writes_trigger);
|
|
|
|
|
ROCKS_LOG_HEADER(
|
|
|
|
|
log, " Options.target_file_size_base: %" PRIu64,
|
|
|
|
|
target_file_size_base);
|
|
|
|
|
ROCKS_LOG_HEADER(log, " Options.target_file_size_multiplier: %d",
|
|
|
|
|
target_file_size_multiplier);
|
|
|
|
|
ROCKS_LOG_HEADER(
|
|
|
|
|
log, " Options.max_bytes_for_level_base: %" PRIu64,
|
|
|
|
|
max_bytes_for_level_base);
|
|
|
|
|
ROCKS_LOG_HEADER(log, "Options.level_compaction_dynamic_level_bytes: %d",
|
|
|
|
|
level_compaction_dynamic_level_bytes);
|
|
|
|
|
ROCKS_LOG_HEADER(log, " Options.max_bytes_for_level_multiplier: %f",
|
|
|
|
|
max_bytes_for_level_multiplier);
|
|
|
|
|
for (size_t i = 0; i < max_bytes_for_level_multiplier_additional.size();
|
|
|
|
|
i++) {
|
|
|
|
|
ROCKS_LOG_HEADER(
|
|
|
|
|
log, "Options.max_bytes_for_level_multiplier_addtl[%" ROCKSDB_PRIszt
|
|
|
|
|
"]: %d",
|
|
|
|
|
i, max_bytes_for_level_multiplier_additional[i]);
|
|
|
|
|
}
|
|
|
|
|
ROCKS_LOG_HEADER(
|
|
|
|
|
log, " Options.max_sequential_skip_in_iterations: %" PRIu64,
|
|
|
|
|
max_sequential_skip_in_iterations);
|
|
|
|
|
ROCKS_LOG_HEADER(
|
|
|
|
|
log, " Options.max_compaction_bytes: %" PRIu64,
|
|
|
|
|
max_compaction_bytes);
|
|
|
|
|
ROCKS_LOG_HEADER(
|
|
|
|
|
log,
|
|
|
|
|
" Options.arena_block_size: %" ROCKSDB_PRIszt,
|
|
|
|
|
arena_block_size);
|
|
|
|
|
ROCKS_LOG_HEADER(log,
|
|
|
|
|
" Options.soft_pending_compaction_bytes_limit: %" PRIu64,
|
|
|
|
|
soft_pending_compaction_bytes_limit);
|
|
|
|
|
ROCKS_LOG_HEADER(log,
|
|
|
|
|
" Options.hard_pending_compaction_bytes_limit: %" PRIu64,
|
|
|
|
|
hard_pending_compaction_bytes_limit);
|
|
|
|
|
ROCKS_LOG_HEADER(log, " Options.rate_limit_delay_max_milliseconds: %u",
|
|
|
|
|
rate_limit_delay_max_milliseconds);
|
|
|
|
|
ROCKS_LOG_HEADER(log, " Options.disable_auto_compactions: %d",
|
|
|
|
|
disable_auto_compactions);
|
|
|
|
|
|
|
|
|
|
const auto& it_compaction_style =
|
|
|
|
|
compaction_style_to_string.find(compaction_style);
|
|
|
|
|
std::string str_compaction_style;
|
|
|
|
|
if (it_compaction_style == compaction_style_to_string.end()) {
|
|
|
|
|
assert(false);
|
|
|
|
|
str_compaction_style = "unknown_" + std::to_string(compaction_style);
|
|
|
|
|
} else {
|
|
|
|
|
str_compaction_style = it_compaction_style->second;
|
|
|
|
|
}
|
|
|
|
|
ROCKS_LOG_HEADER(log,
|
|
|
|
|
" Options.compaction_style: %s",
|
|
|
|
|
str_compaction_style.c_str());
|
|
|
|
|
|
|
|
|
|
const auto& it_compaction_pri =
|
|
|
|
|
compaction_pri_to_string.find(compaction_pri);
|
|
|
|
|
std::string str_compaction_pri;
|
|
|
|
|
if (it_compaction_pri == compaction_pri_to_string.end()) {
|
|
|
|
|
assert(false);
|
|
|
|
|
str_compaction_pri = "unknown_" + std::to_string(compaction_pri);
|
|
|
|
|
} else {
|
|
|
|
|
str_compaction_pri = it_compaction_pri->second;
|
|
|
|
|
}
|
|
|
|
|
ROCKS_LOG_HEADER(log,
|
|
|
|
|
" Options.compaction_pri: %s",
|
|
|
|
|
str_compaction_pri.c_str());
|
|
|
|
|
ROCKS_LOG_HEADER(log,
|
|
|
|
|
"Options.compaction_options_universal.size_ratio: %u",
|
|
|
|
|
compaction_options_universal.size_ratio);
|
|
|
|
|
ROCKS_LOG_HEADER(log,
|
|
|
|
|
"Options.compaction_options_universal.min_merge_width: %u",
|
|
|
|
|
compaction_options_universal.min_merge_width);
|
|
|
|
|
ROCKS_LOG_HEADER(log,
|
|
|
|
|
"Options.compaction_options_universal.max_merge_width: %u",
|
|
|
|
|
compaction_options_universal.max_merge_width);
|
|
|
|
|
ROCKS_LOG_HEADER(
|
|
|
|
|
log,
|
|
|
|
|
"Options.compaction_options_universal."
|
|
|
|
|
"max_size_amplification_percent: %u",
|
|
|
|
|
compaction_options_universal.max_size_amplification_percent);
|
|
|
|
|
ROCKS_LOG_HEADER(
|
|
|
|
|
log,
|
|
|
|
|
"Options.compaction_options_universal.compression_size_percent: %d",
|
|
|
|
|
compaction_options_universal.compression_size_percent);
|
|
|
|
|
const auto& it_compaction_stop_style = compaction_stop_style_to_string.find(
|
|
|
|
|
compaction_options_universal.stop_style);
|
|
|
|
|
std::string str_compaction_stop_style;
|
|
|
|
|
if (it_compaction_stop_style == compaction_stop_style_to_string.end()) {
|
|
|
|
|
assert(false);
|
|
|
|
|
str_compaction_stop_style =
|
|
|
|
|
"unknown_" + std::to_string(compaction_options_universal.stop_style);
|
|
|
|
|
} else {
|
|
|
|
|
str_compaction_stop_style = it_compaction_stop_style->second;
|
|
|
|
|
}
|
|
|
|
|
ROCKS_LOG_HEADER(log,
|
|
|
|
|
"Options.compaction_options_universal.stop_style: %s",
|
|
|
|
|
str_compaction_stop_style.c_str());
|
|
|
|
|
ROCKS_LOG_HEADER(
|
|
|
|
|
log, "Options.compaction_options_fifo.max_table_files_size: %" PRIu64,
|
|
|
|
|
compaction_options_fifo.max_table_files_size);
|
|
|
|
|
ROCKS_LOG_HEADER(log,
|
|
|
|
|
"Options.compaction_options_fifo.allow_compaction: %d",
|
|
|
|
|
compaction_options_fifo.allow_compaction);
|
|
|
|
|
std::ostringstream collector_info;
|
TablePropertiesCollectorFactory
Summary:
This diff addresses task #4296714 and rethinks how users provide us with TablePropertiesCollectors as part of Options.
Here's description of task #4296714:
I'm debugging #4295529 and noticed that our count of user properties kDeletedKeys is wrong. We're sharing one single InternalKeyPropertiesCollector with all Table Builders. In LOG Files, we're outputting number of kDeletedKeys as connected with a single table, while it's actually the total count of deleted keys since creation of the DB.
For example, this table has 3155 entries and 1391828 deleted keys.
The problem with current approach that we call methods on a single TablePropertiesCollector for all the tables we create. Even worse, we could do it from multiple threads at the same time and TablePropertiesCollector has no way of knowing which table we're calling it for.
Good part: Looks like nobody inside Facebook is using Options::table_properties_collectors. This means we should be able to painfully change the API.
In this change, I introduce TablePropertiesCollectorFactory. For every table we create, we call `CreateTablePropertiesCollector`, which creates a TablePropertiesCollector for a single table. We then use it sequentially from a single thread, which means it doesn't have to be thread-safe.
Test Plan:
Added a test in table_properties_collector_test that fails on master (build two tables, assert that kDeletedKeys count is correct for the second one).
Also, all other tests
Reviewers: sdong, dhruba, haobo, kailiu
Reviewed By: kailiu
CC: leveldb
Differential Revision: https://reviews.facebook.net/D18579
11 years ago
|
|
|
for (const auto& collector_factory : table_properties_collector_factories) {
|
|
|
|
|
collector_info << collector_factory->ToString() << ';';
|
|
|
|
|
}
|
|
|
|
|
ROCKS_LOG_HEADER(
|
|
|
|
|
log, " Options.table_properties_collectors: %s",
|
|
|
|
|
collector_info.str().c_str());
|
|
|
|
|
ROCKS_LOG_HEADER(log,
|
|
|
|
|
" Options.inplace_update_support: %d",
|
|
|
|
|
inplace_update_support);
|
|
|
|
|
ROCKS_LOG_HEADER(
|
|
|
|
|
log,
|
|
|
|
|
" Options.inplace_update_num_locks: %" ROCKSDB_PRIszt,
|
|
|
|
|
inplace_update_num_locks);
|
|
|
|
|
// TODO: easier config for bloom (maybe based on avg key/value size)
|
|
|
|
|
ROCKS_LOG_HEADER(
|
|
|
|
|
log, " Options.memtable_prefix_bloom_size_ratio: %f",
|
|
|
|
|
memtable_prefix_bloom_size_ratio);
|
|
|
|
|
ROCKS_LOG_HEADER(log,
|
|
|
|
|
" Options.memtable_whole_key_filtering: %d",
|
|
|
|
|
memtable_whole_key_filtering);
|
|
|
|
|
|
|
|
|
|
ROCKS_LOG_HEADER(log, " Options.memtable_huge_page_size: %" ROCKSDB_PRIszt,
|
|
|
|
|
memtable_huge_page_size);
|
|
|
|
|
ROCKS_LOG_HEADER(log,
|
|
|
|
|
" Options.bloom_locality: %d",
|
|
|
|
|
bloom_locality);
|
|
|
|
|
|
|
|
|
|
ROCKS_LOG_HEADER(
|
|
|
|
|
log,
|
|
|
|
|
" Options.max_successive_merges: %" ROCKSDB_PRIszt,
|
|
|
|
|
max_successive_merges);
|
|
|
|
|
ROCKS_LOG_HEADER(log,
|
|
|
|
|
" Options.optimize_filters_for_hits: %d",
|
|
|
|
|
optimize_filters_for_hits);
|
|
|
|
|
ROCKS_LOG_HEADER(log, " Options.paranoid_file_checks: %d",
|
|
|
|
|
paranoid_file_checks);
|
|
|
|
|
ROCKS_LOG_HEADER(log, " Options.force_consistency_checks: %d",
|
|
|
|
|
force_consistency_checks);
|
|
|
|
|
ROCKS_LOG_HEADER(log, " Options.report_bg_io_stats: %d",
|
|
|
|
|
report_bg_io_stats);
|
|
|
|
|
ROCKS_LOG_HEADER(log, " Options.ttl: %" PRIu64,
|
|
|
|
|
ttl);
|
Periodic Compactions (#5166)
Summary:
Introducing Periodic Compactions.
This feature allows all the files in a CF to be periodically compacted. It could help in catching any corruptions that could creep into the DB proactively as every file is constantly getting re-compacted. And also, of course, it helps to cleanup data older than certain threshold.
- Introduced a new option `periodic_compaction_time` to control how long a file can live without being compacted in a CF.
- This works across all levels.
- The files are put in the same level after going through the compaction. (Related files in the same level are picked up as `ExpandInputstoCleanCut` is used).
- Compaction filters, if any, are invoked as usual.
- A new table property, `file_creation_time`, is introduced to implement this feature. This property is set to the time at which the SST file was created (and that time is given by the underlying Env/OS).
This feature can be enabled on its own, or in conjunction with `ttl`. It is possible to set a different time threshold for the bottom level when used in conjunction with ttl. Since `ttl` works only on 0 to last but one levels, you could set `ttl` to, say, 1 day, and `periodic_compaction_time` to, say, 7 days. Since `ttl < periodic_compaction_time` all files in last but one levels keep getting picked up based on ttl, and almost never based on periodic_compaction_time. The files in the bottom level get picked up for compaction based on `periodic_compaction_time`.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5166
Differential Revision: D14884441
Pulled By: sagar0
fbshipit-source-id: 408426cbacb409c06386a98632dcf90bfa1bda47
6 years ago
|
|
|
ROCKS_LOG_HEADER(log,
|
|
|
|
|
" Options.periodic_compaction_seconds: %" PRIu64,
|
|
|
|
|
periodic_compaction_seconds);
|
Make it possible to force the garbage collection of the oldest blob files (#8994)
Summary:
The current BlobDB garbage collection logic works by relocating the valid
blobs from the oldest blob files as they are encountered during compaction,
and cleaning up blob files once they contain nothing but garbage. However,
with sufficiently skewed workloads, it is theoretically possible to end up in a
situation when few or no compactions get scheduled for the SST files that contain
references to the oldest blob files, which can lead to increased space amp due
to the lack of GC.
In order to efficiently handle such workloads, the patch adds a new BlobDB
configuration option called `blob_garbage_collection_force_threshold`,
which signals to BlobDB to schedule targeted compactions for the SST files
that keep alive the oldest batch of blob files if the overall ratio of garbage in
the given blob files meets the threshold *and* all the given blob files are
eligible for GC based on `blob_garbage_collection_age_cutoff`. (For example,
if the new option is set to 0.9, targeted compactions will get scheduled if the
sum of garbage bytes meets or exceeds 90% of the sum of total bytes in the
oldest blob files, assuming all affected blob files are below the age-based cutoff.)
The net result of these targeted compactions is that the valid blobs in the oldest
blob files are relocated and the oldest blob files themselves cleaned up (since
*all* SST files that rely on them get compacted away).
These targeted compactions are similar to periodic compactions in the sense
that they force certain SST files that otherwise would not get picked up to undergo
compaction and also in the sense that instead of merging files from multiple levels,
they target a single file. (Note: such compactions might still include neighboring files
from the same level due to the need of having a "clean cut" boundary but they never
include any files from any other level.)
This functionality is currently only supported with the leveled compaction style
and is inactive by default (since the default value is set to 1.0, i.e. 100%).
Pull Request resolved: https://github.com/facebook/rocksdb/pull/8994
Test Plan: Ran `make check` and tested using `db_bench` and the stress/crash tests.
Reviewed By: riversand963
Differential Revision: D31489850
Pulled By: ltamasi
fbshipit-source-id: 44057d511726a0e2a03c5d9313d7511b3f0c4eab
3 years ago
|
|
|
ROCKS_LOG_HEADER(log, " Options.enable_blob_files: %s",
|
|
|
|
|
enable_blob_files ? "true" : "false");
|
Make it possible to force the garbage collection of the oldest blob files (#8994)
Summary:
The current BlobDB garbage collection logic works by relocating the valid
blobs from the oldest blob files as they are encountered during compaction,
and cleaning up blob files once they contain nothing but garbage. However,
with sufficiently skewed workloads, it is theoretically possible to end up in a
situation when few or no compactions get scheduled for the SST files that contain
references to the oldest blob files, which can lead to increased space amp due
to the lack of GC.
In order to efficiently handle such workloads, the patch adds a new BlobDB
configuration option called `blob_garbage_collection_force_threshold`,
which signals to BlobDB to schedule targeted compactions for the SST files
that keep alive the oldest batch of blob files if the overall ratio of garbage in
the given blob files meets the threshold *and* all the given blob files are
eligible for GC based on `blob_garbage_collection_age_cutoff`. (For example,
if the new option is set to 0.9, targeted compactions will get scheduled if the
sum of garbage bytes meets or exceeds 90% of the sum of total bytes in the
oldest blob files, assuming all affected blob files are below the age-based cutoff.)
The net result of these targeted compactions is that the valid blobs in the oldest
blob files are relocated and the oldest blob files themselves cleaned up (since
*all* SST files that rely on them get compacted away).
These targeted compactions are similar to periodic compactions in the sense
that they force certain SST files that otherwise would not get picked up to undergo
compaction and also in the sense that instead of merging files from multiple levels,
they target a single file. (Note: such compactions might still include neighboring files
from the same level due to the need of having a "clean cut" boundary but they never
include any files from any other level.)
This functionality is currently only supported with the leveled compaction style
and is inactive by default (since the default value is set to 1.0, i.e. 100%).
Pull Request resolved: https://github.com/facebook/rocksdb/pull/8994
Test Plan: Ran `make check` and tested using `db_bench` and the stress/crash tests.
Reviewed By: riversand963
Differential Revision: D31489850
Pulled By: ltamasi
fbshipit-source-id: 44057d511726a0e2a03c5d9313d7511b3f0c4eab
3 years ago
|
|
|
ROCKS_LOG_HEADER(
|
|
|
|
|
log, " Options.min_blob_size: %" PRIu64,
|
|
|
|
|
min_blob_size);
|
|
|
|
|
ROCKS_LOG_HEADER(
|
|
|
|
|
log, " Options.blob_file_size: %" PRIu64,
|
|
|
|
|
blob_file_size);
|
|
|
|
|
ROCKS_LOG_HEADER(log, " Options.blob_compression_type: %s",
|
|
|
|
|
CompressionTypeToString(blob_compression_type).c_str());
|
Make it possible to force the garbage collection of the oldest blob files (#8994)
Summary:
The current BlobDB garbage collection logic works by relocating the valid
blobs from the oldest blob files as they are encountered during compaction,
and cleaning up blob files once they contain nothing but garbage. However,
with sufficiently skewed workloads, it is theoretically possible to end up in a
situation when few or no compactions get scheduled for the SST files that contain
references to the oldest blob files, which can lead to increased space amp due
to the lack of GC.
In order to efficiently handle such workloads, the patch adds a new BlobDB
configuration option called `blob_garbage_collection_force_threshold`,
which signals to BlobDB to schedule targeted compactions for the SST files
that keep alive the oldest batch of blob files if the overall ratio of garbage in
the given blob files meets the threshold *and* all the given blob files are
eligible for GC based on `blob_garbage_collection_age_cutoff`. (For example,
if the new option is set to 0.9, targeted compactions will get scheduled if the
sum of garbage bytes meets or exceeds 90% of the sum of total bytes in the
oldest blob files, assuming all affected blob files are below the age-based cutoff.)
The net result of these targeted compactions is that the valid blobs in the oldest
blob files are relocated and the oldest blob files themselves cleaned up (since
*all* SST files that rely on them get compacted away).
These targeted compactions are similar to periodic compactions in the sense
that they force certain SST files that otherwise would not get picked up to undergo
compaction and also in the sense that instead of merging files from multiple levels,
they target a single file. (Note: such compactions might still include neighboring files
from the same level due to the need of having a "clean cut" boundary but they never
include any files from any other level.)
This functionality is currently only supported with the leveled compaction style
and is inactive by default (since the default value is set to 1.0, i.e. 100%).
Pull Request resolved: https://github.com/facebook/rocksdb/pull/8994
Test Plan: Ran `make check` and tested using `db_bench` and the stress/crash tests.
Reviewed By: riversand963
Differential Revision: D31489850
Pulled By: ltamasi
fbshipit-source-id: 44057d511726a0e2a03c5d9313d7511b3f0c4eab
3 years ago
|
|
|
ROCKS_LOG_HEADER(log, " Options.enable_blob_garbage_collection: %s",
|
|
|
|
|
enable_blob_garbage_collection ? "true" : "false");
|
Make it possible to force the garbage collection of the oldest blob files (#8994)
Summary:
The current BlobDB garbage collection logic works by relocating the valid
blobs from the oldest blob files as they are encountered during compaction,
and cleaning up blob files once they contain nothing but garbage. However,
with sufficiently skewed workloads, it is theoretically possible to end up in a
situation when few or no compactions get scheduled for the SST files that contain
references to the oldest blob files, which can lead to increased space amp due
to the lack of GC.
In order to efficiently handle such workloads, the patch adds a new BlobDB
configuration option called `blob_garbage_collection_force_threshold`,
which signals to BlobDB to schedule targeted compactions for the SST files
that keep alive the oldest batch of blob files if the overall ratio of garbage in
the given blob files meets the threshold *and* all the given blob files are
eligible for GC based on `blob_garbage_collection_age_cutoff`. (For example,
if the new option is set to 0.9, targeted compactions will get scheduled if the
sum of garbage bytes meets or exceeds 90% of the sum of total bytes in the
oldest blob files, assuming all affected blob files are below the age-based cutoff.)
The net result of these targeted compactions is that the valid blobs in the oldest
blob files are relocated and the oldest blob files themselves cleaned up (since
*all* SST files that rely on them get compacted away).
These targeted compactions are similar to periodic compactions in the sense
that they force certain SST files that otherwise would not get picked up to undergo
compaction and also in the sense that instead of merging files from multiple levels,
they target a single file. (Note: such compactions might still include neighboring files
from the same level due to the need of having a "clean cut" boundary but they never
include any files from any other level.)
This functionality is currently only supported with the leveled compaction style
and is inactive by default (since the default value is set to 1.0, i.e. 100%).
Pull Request resolved: https://github.com/facebook/rocksdb/pull/8994
Test Plan: Ran `make check` and tested using `db_bench` and the stress/crash tests.
Reviewed By: riversand963
Differential Revision: D31489850
Pulled By: ltamasi
fbshipit-source-id: 44057d511726a0e2a03c5d9313d7511b3f0c4eab
3 years ago
|
|
|
ROCKS_LOG_HEADER(log, " Options.blob_garbage_collection_age_cutoff: %f",
|
|
|
|
|
blob_garbage_collection_age_cutoff);
|
Make it possible to force the garbage collection of the oldest blob files (#8994)
Summary:
The current BlobDB garbage collection logic works by relocating the valid
blobs from the oldest blob files as they are encountered during compaction,
and cleaning up blob files once they contain nothing but garbage. However,
with sufficiently skewed workloads, it is theoretically possible to end up in a
situation when few or no compactions get scheduled for the SST files that contain
references to the oldest blob files, which can lead to increased space amp due
to the lack of GC.
In order to efficiently handle such workloads, the patch adds a new BlobDB
configuration option called `blob_garbage_collection_force_threshold`,
which signals to BlobDB to schedule targeted compactions for the SST files
that keep alive the oldest batch of blob files if the overall ratio of garbage in
the given blob files meets the threshold *and* all the given blob files are
eligible for GC based on `blob_garbage_collection_age_cutoff`. (For example,
if the new option is set to 0.9, targeted compactions will get scheduled if the
sum of garbage bytes meets or exceeds 90% of the sum of total bytes in the
oldest blob files, assuming all affected blob files are below the age-based cutoff.)
The net result of these targeted compactions is that the valid blobs in the oldest
blob files are relocated and the oldest blob files themselves cleaned up (since
*all* SST files that rely on them get compacted away).
These targeted compactions are similar to periodic compactions in the sense
that they force certain SST files that otherwise would not get picked up to undergo
compaction and also in the sense that instead of merging files from multiple levels,
they target a single file. (Note: such compactions might still include neighboring files
from the same level due to the need of having a "clean cut" boundary but they never
include any files from any other level.)
This functionality is currently only supported with the leveled compaction style
and is inactive by default (since the default value is set to 1.0, i.e. 100%).
Pull Request resolved: https://github.com/facebook/rocksdb/pull/8994
Test Plan: Ran `make check` and tested using `db_bench` and the stress/crash tests.
Reviewed By: riversand963
Differential Revision: D31489850
Pulled By: ltamasi
fbshipit-source-id: 44057d511726a0e2a03c5d9313d7511b3f0c4eab
3 years ago
|
|
|
ROCKS_LOG_HEADER(log, "Options.blob_garbage_collection_force_threshold: %f",
|
|
|
|
|
blob_garbage_collection_force_threshold);
|
|
|
|
|
ROCKS_LOG_HEADER(
|
|
|
|
|
log, " Options.blob_compaction_readahead_size: %" PRIu64,
|
|
|
|
|
blob_compaction_readahead_size);
|
|
|
|
|
} // ColumnFamilyOptions::Dump
|
|
|
|
|
|
|
|
|
|
void Options::Dump(Logger* log) const {
|
|
|
|
|
DBOptions::Dump(log);
|
|
|
|
|
ColumnFamilyOptions::Dump(log);
|
|
|
|
|
} // Options::Dump
|
|
|
|
|
|
|
|
|
|
void Options::DumpCFOptions(Logger* log) const {
|
|
|
|
|
ColumnFamilyOptions::Dump(log);
|
|
|
|
|
} // Options::DumpCFOptions
|
|
|
|
|
|
|
|
|
|
//
|
|
|
|
|
// The goal of this method is to create a configuration that
|
|
|
|
|
// allows an application to write all files into L0 and
|
|
|
|
|
// then do a single compaction to output all files into L1.
|
|
|
|
|
Options*
|
|
|
|
|
Options::PrepareForBulkLoad()
|
|
|
|
|
{
|
|
|
|
|
// never slowdown ingest.
|
|
|
|
|
level0_file_num_compaction_trigger = (1<<30);
|
|
|
|
|
level0_slowdown_writes_trigger = (1<<30);
|
|
|
|
|
level0_stop_writes_trigger = (1<<30);
|
|
|
|
|
soft_pending_compaction_bytes_limit = 0;
|
|
|
|
|
hard_pending_compaction_bytes_limit = 0;
|
|
|
|
|
|
|
|
|
|
// no auto compactions please. The application should issue a
|
|
|
|
|
// manual compaction after all data is loaded into L0.
|
|
|
|
|
disable_auto_compactions = true;
|
|
|
|
|
// A manual compaction run should pick all files in L0 in
|
|
|
|
|
// a single compaction run.
|
|
|
|
|
max_compaction_bytes = (static_cast<uint64_t>(1) << 60);
|
|
|
|
|
|
|
|
|
|
// It is better to have only 2 levels, otherwise a manual
|
|
|
|
|
// compaction would compact at every possible level, thereby
|
|
|
|
|
// increasing the total time needed for compactions.
|
|
|
|
|
num_levels = 2;
|
|
|
|
|
|
|
|
|
|
// Need to allow more write buffers to allow more parallism
|
|
|
|
|
// of flushes.
|
|
|
|
|
max_write_buffer_number = 6;
|
|
|
|
|
min_write_buffer_number_to_merge = 1;
|
|
|
|
|
|
|
|
|
|
// When compaction is disabled, more parallel flush threads can
|
|
|
|
|
// help with write throughput.
|
|
|
|
|
max_background_flushes = 4;
|
|
|
|
|
|
|
|
|
|
// Prevent a memtable flush to automatically promote files
|
|
|
|
|
// to L1. This is helpful so that all files that are
|
|
|
|
|
// input to the manual compaction are all at L0.
|
|
|
|
|
max_background_compactions = 2;
|
|
|
|
|
|
|
|
|
|
// The compaction would create large files in L1.
|
|
|
|
|
target_file_size_base = 256 * 1024 * 1024;
|
|
|
|
|
return this;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
Options* Options::OptimizeForSmallDb() {
|
|
|
|
|
// 16MB block cache
|
|
|
|
|
std::shared_ptr<Cache> cache = NewLRUCache(16 << 20);
|
|
|
|
|
|
|
|
|
|
ColumnFamilyOptions::OptimizeForSmallDb(&cache);
|
|
|
|
|
DBOptions::OptimizeForSmallDb(&cache);
|
|
|
|
|
return this;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
Options* Options::DisableExtraChecks() {
|
|
|
|
|
// See https://github.com/facebook/rocksdb/issues/9354
|
|
|
|
|
force_consistency_checks = false;
|
|
|
|
|
// Considered but no clear performance impact seen:
|
|
|
|
|
// * check_flush_compaction_key_order
|
|
|
|
|
// * paranoid_checks
|
|
|
|
|
// * flush_verify_memtable_count
|
|
|
|
|
// By current API contract, not including
|
|
|
|
|
// * verify_checksums
|
|
|
|
|
// because checking storage data integrity is a more standard practice.
|
|
|
|
|
return this;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
Options* Options::OldDefaults(int rocksdb_major_version,
|
|
|
|
|
int rocksdb_minor_version) {
|
|
|
|
|
ColumnFamilyOptions::OldDefaults(rocksdb_major_version,
|
|
|
|
|
rocksdb_minor_version);
|
|
|
|
|
DBOptions::OldDefaults(rocksdb_major_version, rocksdb_minor_version);
|
|
|
|
|
return this;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
DBOptions* DBOptions::OldDefaults(int rocksdb_major_version,
|
|
|
|
|
int rocksdb_minor_version) {
|
|
|
|
|
if (rocksdb_major_version < 4 ||
|
|
|
|
|
(rocksdb_major_version == 4 && rocksdb_minor_version < 7)) {
|
|
|
|
|
max_file_opening_threads = 1;
|
|
|
|
|
table_cache_numshardbits = 4;
|
|
|
|
|
}
|
|
|
|
|
if (rocksdb_major_version < 5 ||
|
|
|
|
|
(rocksdb_major_version == 5 && rocksdb_minor_version < 2)) {
|
|
|
|
|
delayed_write_rate = 2 * 1024U * 1024U;
|
|
|
|
|
} else if (rocksdb_major_version < 5 ||
|
|
|
|
|
(rocksdb_major_version == 5 && rocksdb_minor_version < 6)) {
|
|
|
|
|
delayed_write_rate = 16 * 1024U * 1024U;
|
|
|
|
|
}
|
|
|
|
|
max_open_files = 5000;
|
|
|
|
|
wal_recovery_mode = WALRecoveryMode::kTolerateCorruptedTailRecords;
|
|
|
|
|
return this;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
ColumnFamilyOptions* ColumnFamilyOptions::OldDefaults(
|
|
|
|
|
int rocksdb_major_version, int rocksdb_minor_version) {
|
|
|
|
|
if (rocksdb_major_version < 5 ||
|
|
|
|
|
(rocksdb_major_version == 5 && rocksdb_minor_version <= 18)) {
|
|
|
|
|
compaction_pri = CompactionPri::kByCompensatedSize;
|
|
|
|
|
}
|
|
|
|
|
if (rocksdb_major_version < 4 ||
|
|
|
|
|
(rocksdb_major_version == 4 && rocksdb_minor_version < 7)) {
|
|
|
|
|
write_buffer_size = 4 << 20;
|
|
|
|
|
target_file_size_base = 2 * 1048576;
|
|
|
|
|
max_bytes_for_level_base = 10 * 1048576;
|
|
|
|
|
soft_pending_compaction_bytes_limit = 0;
|
|
|
|
|
hard_pending_compaction_bytes_limit = 0;
|
|
|
|
|
}
|
|
|
|
|
if (rocksdb_major_version < 5) {
|
|
|
|
|
level0_stop_writes_trigger = 24;
|
|
|
|
|
} else if (rocksdb_major_version == 5 && rocksdb_minor_version < 2) {
|
|
|
|
|
level0_stop_writes_trigger = 30;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return this;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Optimization functions
|
|
|
|
|
DBOptions* DBOptions::OptimizeForSmallDb(std::shared_ptr<Cache>* cache) {
|
|
|
|
|
max_file_opening_threads = 1;
|
|
|
|
|
max_open_files = 5000;
|
|
|
|
|
|
|
|
|
|
// Cost memtable to block cache too.
|
|
|
|
|
std::shared_ptr<ROCKSDB_NAMESPACE::WriteBufferManager> wbm =
|
|
|
|
|
std::make_shared<ROCKSDB_NAMESPACE::WriteBufferManager>(
|
|
|
|
|
0, (cache != nullptr) ? *cache : std::shared_ptr<Cache>());
|
|
|
|
|
write_buffer_manager = wbm;
|
|
|
|
|
|
|
|
|
|
return this;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
ColumnFamilyOptions* ColumnFamilyOptions::OptimizeForSmallDb(
|
|
|
|
|
std::shared_ptr<Cache>* cache) {
|
|
|
|
|
write_buffer_size = 2 << 20;
|
|
|
|
|
target_file_size_base = 2 * 1048576;
|
|
|
|
|
max_bytes_for_level_base = 10 * 1048576;
|
|
|
|
|
soft_pending_compaction_bytes_limit = 256 * 1048576;
|
|
|
|
|
hard_pending_compaction_bytes_limit = 1073741824ul;
|
|
|
|
|
|
|
|
|
|
BlockBasedTableOptions table_options;
|
|
|
|
|
table_options.block_cache =
|
|
|
|
|
(cache != nullptr) ? *cache : std::shared_ptr<Cache>();
|
|
|
|
|
table_options.cache_index_and_filter_blocks = true;
|
|
|
|
|
// Two level iterator to avoid LRU cache imbalance
|
|
|
|
|
table_options.index_type =
|
|
|
|
|
BlockBasedTableOptions::IndexType::kTwoLevelIndexSearch;
|
|
|
|
|
table_factory.reset(new BlockBasedTableFactory(table_options));
|
|
|
|
|
|
|
|
|
|
return this;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#ifndef ROCKSDB_LITE
|
|
|
|
|
ColumnFamilyOptions* ColumnFamilyOptions::OptimizeForPointLookup(
|
|
|
|
|
uint64_t block_cache_size_mb) {
|
|
|
|
|
BlockBasedTableOptions block_based_options;
|
|
|
|
|
block_based_options.data_block_index_type =
|
|
|
|
|
BlockBasedTableOptions::kDataBlockBinaryAndHash;
|
|
|
|
|
block_based_options.data_block_hash_table_util_ratio = 0.75;
|
|
|
|
|
block_based_options.filter_policy.reset(NewBloomFilterPolicy(10));
|
|
|
|
|
block_based_options.block_cache =
|
|
|
|
|
NewLRUCache(static_cast<size_t>(block_cache_size_mb * 1024 * 1024));
|
|
|
|
|
table_factory.reset(new BlockBasedTableFactory(block_based_options));
|
|
|
|
|
memtable_prefix_bloom_size_ratio = 0.02;
|
|
|
|
|
memtable_whole_key_filtering = true;
|
|
|
|
|
return this;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
ColumnFamilyOptions* ColumnFamilyOptions::OptimizeLevelStyleCompaction(
|
|
|
|
|
uint64_t memtable_memory_budget) {
|
|
|
|
|
write_buffer_size = static_cast<size_t>(memtable_memory_budget / 4);
|
|
|
|
|
// merge two memtables when flushing to L0
|
|
|
|
|
min_write_buffer_number_to_merge = 2;
|
|
|
|
|
// this means we'll use 50% extra memory in the worst case, but will reduce
|
|
|
|
|
// write stalls.
|
|
|
|
|
max_write_buffer_number = 6;
|
|
|
|
|
// start flushing L0->L1 as soon as possible. each file on level0 is
|
|
|
|
|
// (memtable_memory_budget / 2). This will flush level 0 when it's bigger than
|
|
|
|
|
// memtable_memory_budget.
|
|
|
|
|
level0_file_num_compaction_trigger = 2;
|
|
|
|
|
// doesn't really matter much, but we don't want to create too many files
|
|
|
|
|
target_file_size_base = memtable_memory_budget / 8;
|
|
|
|
|
// make Level1 size equal to Level0 size, so that L0->L1 compactions are fast
|
|
|
|
|
max_bytes_for_level_base = memtable_memory_budget;
|
|
|
|
|
|
|
|
|
|
// level style compaction
|
|
|
|
|
compaction_style = kCompactionStyleLevel;
|
|
|
|
|
|
|
|
|
|
// only compress levels >= 2
|
|
|
|
|
compression_per_level.resize(num_levels);
|
|
|
|
|
for (int i = 0; i < num_levels; ++i) {
|
|
|
|
|
if (i < 2) {
|
|
|
|
|
compression_per_level[i] = kNoCompression;
|
|
|
|
|
} else {
|
|
|
|
|
compression_per_level[i] =
|
|
|
|
|
LZ4_Supported()
|
|
|
|
|
? kLZ4Compression
|
|
|
|
|
: (Snappy_Supported() ? kSnappyCompression : kNoCompression);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
return this;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
ColumnFamilyOptions* ColumnFamilyOptions::OptimizeUniversalStyleCompaction(
|
|
|
|
|
uint64_t memtable_memory_budget) {
|
|
|
|
|
write_buffer_size = static_cast<size_t>(memtable_memory_budget / 4);
|
|
|
|
|
// merge two memtables when flushing to L0
|
|
|
|
|
min_write_buffer_number_to_merge = 2;
|
|
|
|
|
// this means we'll use 50% extra memory in the worst case, but will reduce
|
|
|
|
|
// write stalls.
|
|
|
|
|
max_write_buffer_number = 6;
|
|
|
|
|
// universal style compaction
|
|
|
|
|
compaction_style = kCompactionStyleUniversal;
|
|
|
|
|
compaction_options_universal.compression_size_percent = 80;
|
|
|
|
|
return this;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
DBOptions* DBOptions::IncreaseParallelism(int total_threads) {
|
|
|
|
|
max_background_jobs = total_threads;
|
|
|
|
|
env->SetBackgroundThreads(total_threads, Env::LOW);
|
|
|
|
|
env->SetBackgroundThreads(1, Env::HIGH);
|
|
|
|
|
return this;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#endif // !ROCKSDB_LITE
|
|
|
|
|
|
|
|
|
|
ReadOptions::ReadOptions()
|
|
|
|
|
: snapshot(nullptr),
|
|
|
|
|
iterate_lower_bound(nullptr),
|
|
|
|
|
iterate_upper_bound(nullptr),
|
|
|
|
|
readahead_size(0),
|
|
|
|
|
max_skippable_internal_keys(0),
|
|
|
|
|
read_tier(kReadAllTier),
|
|
|
|
|
verify_checksums(true),
|
|
|
|
|
fill_cache(true),
|
|
|
|
|
tailing(false),
|
|
|
|
|
managed(false),
|
|
|
|
|
total_order_seek(false),
|
|
|
|
|
auto_prefix_mode(false),
|
|
|
|
|
prefix_same_as_start(false),
|
|
|
|
|
pin_data(false),
|
|
|
|
|
background_purge_on_iterator_cleanup(false),
|
Added support for differential snapshots
Summary:
The motivation for this PR is to add to RocksDB support for differential (incremental) snapshots, as snapshot of the DB changes between two points in time (one can think of it as diff between to sequence numbers, or the diff D which can be thought of as an SST file or just set of KVs that can be applied to sequence number S1 to get the database to the state at sequence number S2).
This feature would be useful for various distributed storages layers built on top of RocksDB, as it should help reduce resources (time and network bandwidth) needed to recover and rebuilt DB instances as replicas in the context of distributed storages.
From the API standpoint that would like client app requesting iterator between (start seqnum) and current DB state, and reading the "diff".
This is a very draft PR for initial review in the discussion on the approach, i'm going to rework some parts and keep updating the PR.
For now, what's done here according to initial discussions:
Preserving deletes:
- We want to be able to optionally preserve recent deletes for some defined period of time, so that if a delete came in recently and might need to be included in the next incremental snapshot it would't get dropped by a compaction. This is done by adding new param to Options (preserve deletes flag) and new variable to DB Impl where we keep track of the sequence number after which we don't want to drop tombstones, even if they are otherwise eligible for deletion.
- I also added a new API call for clients to be able to advance this cutoff seqnum after which we drop deletes; i assume it's more flexible to let clients control this, since otherwise we'd need to keep some kind of timestamp < -- > seqnum mapping inside the DB, which sounds messy and painful to support. Clients could make use of it by periodically calling GetLatestSequenceNumber(), noting the timestamp, doing some calculation and figuring out by how much we need to advance the cutoff seqnum.
- Compaction codepath in compaction_iterator.cc has been modified to avoid dropping tombstones with seqnum > cutoff seqnum.
Iterator changes:
- couple params added to ReadOptions, to optionally allow client to request internal keys instead of user keys (so that client can get the latest value of a key, be it delete marker or a put), as well as min timestamp and min seqnum.
TableCache changes:
- I modified table_cache code to be able to quickly exclude SST files from iterators heep if creation_time on the file is less then iter_start_ts as passed in ReadOptions. That would help a lot in some DB settings (like reading very recent data only or using FIFO compactions), but not so much for universal compaction with more or less long iterator time span.
What's left:
- Still looking at how to best plug that inside DBIter codepath. So far it seems that FindNextUserKeyInternal only parses values as UserKeys, and iter->key() call generally returns user key. Can we add new API to DBIter as internal_key(), and modify this internal method to optionally set saved_key_ to point to the full internal key? I don't need to store actual seqnum there, but I do need to store type.
Closes https://github.com/facebook/rocksdb/pull/2999
Differential Revision: D6175602
Pulled By: mikhail-antonov
fbshipit-source-id: c779a6696ee2d574d86c69cec866a3ae095aa900
7 years ago
|
|
|
ignore_range_deletions(false),
|
|
|
|
|
iter_start_seqnum(0),
|
|
|
|
|
timestamp(nullptr),
|
|
|
|
|
iter_start_ts(nullptr),
|
|
|
|
|
deadline(std::chrono::microseconds::zero()),
|
|
|
|
|
io_timeout(std::chrono::microseconds::zero()),
|
|
|
|
|
value_size_soft_limit(std::numeric_limits<uint64_t>::max()),
|
|
|
|
|
adaptive_readahead(false) {}
|
|
|
|
|
|
|
|
|
|
ReadOptions::ReadOptions(bool cksum, bool cache)
|
|
|
|
|
: snapshot(nullptr),
|
|
|
|
|
iterate_lower_bound(nullptr),
|
|
|
|
|
iterate_upper_bound(nullptr),
|
|
|
|
|
readahead_size(0),
|
|
|
|
|
max_skippable_internal_keys(0),
|
|
|
|
|
read_tier(kReadAllTier),
|
|
|
|
|
verify_checksums(cksum),
|
|
|
|
|
fill_cache(cache),
|
|
|
|
|
tailing(false),
|
|
|
|
|
managed(false),
|
|
|
|
|
total_order_seek(false),
|
|
|
|
|
auto_prefix_mode(false),
|
|
|
|
|
prefix_same_as_start(false),
|
|
|
|
|
pin_data(false),
|
|
|
|
|
background_purge_on_iterator_cleanup(false),
|
Added support for differential snapshots
Summary:
The motivation for this PR is to add to RocksDB support for differential (incremental) snapshots, as snapshot of the DB changes between two points in time (one can think of it as diff between to sequence numbers, or the diff D which can be thought of as an SST file or just set of KVs that can be applied to sequence number S1 to get the database to the state at sequence number S2).
This feature would be useful for various distributed storages layers built on top of RocksDB, as it should help reduce resources (time and network bandwidth) needed to recover and rebuilt DB instances as replicas in the context of distributed storages.
From the API standpoint that would like client app requesting iterator between (start seqnum) and current DB state, and reading the "diff".
This is a very draft PR for initial review in the discussion on the approach, i'm going to rework some parts and keep updating the PR.
For now, what's done here according to initial discussions:
Preserving deletes:
- We want to be able to optionally preserve recent deletes for some defined period of time, so that if a delete came in recently and might need to be included in the next incremental snapshot it would't get dropped by a compaction. This is done by adding new param to Options (preserve deletes flag) and new variable to DB Impl where we keep track of the sequence number after which we don't want to drop tombstones, even if they are otherwise eligible for deletion.
- I also added a new API call for clients to be able to advance this cutoff seqnum after which we drop deletes; i assume it's more flexible to let clients control this, since otherwise we'd need to keep some kind of timestamp < -- > seqnum mapping inside the DB, which sounds messy and painful to support. Clients could make use of it by periodically calling GetLatestSequenceNumber(), noting the timestamp, doing some calculation and figuring out by how much we need to advance the cutoff seqnum.
- Compaction codepath in compaction_iterator.cc has been modified to avoid dropping tombstones with seqnum > cutoff seqnum.
Iterator changes:
- couple params added to ReadOptions, to optionally allow client to request internal keys instead of user keys (so that client can get the latest value of a key, be it delete marker or a put), as well as min timestamp and min seqnum.
TableCache changes:
- I modified table_cache code to be able to quickly exclude SST files from iterators heep if creation_time on the file is less then iter_start_ts as passed in ReadOptions. That would help a lot in some DB settings (like reading very recent data only or using FIFO compactions), but not so much for universal compaction with more or less long iterator time span.
What's left:
- Still looking at how to best plug that inside DBIter codepath. So far it seems that FindNextUserKeyInternal only parses values as UserKeys, and iter->key() call generally returns user key. Can we add new API to DBIter as internal_key(), and modify this internal method to optionally set saved_key_ to point to the full internal key? I don't need to store actual seqnum there, but I do need to store type.
Closes https://github.com/facebook/rocksdb/pull/2999
Differential Revision: D6175602
Pulled By: mikhail-antonov
fbshipit-source-id: c779a6696ee2d574d86c69cec866a3ae095aa900
7 years ago
|
|
|
ignore_range_deletions(false),
|
|
|
|
|
iter_start_seqnum(0),
|
|
|
|
|
timestamp(nullptr),
|
|
|
|
|
iter_start_ts(nullptr),
|
|
|
|
|
deadline(std::chrono::microseconds::zero()),
|
|
|
|
|
io_timeout(std::chrono::microseconds::zero()),
|
|
|
|
|
value_size_soft_limit(std::numeric_limits<uint64_t>::max()),
|
|
|
|
|
adaptive_readahead(false) {}
|
|
|
|
|
|
|
|
|
|
} // namespace ROCKSDB_NAMESPACE
|
