Store DB identity and DB session ID in SST files (#6983)

Summary:
`db_id` and `db_session_id` are now part of the table properties for all formats and stored in SST files. This adds about 99 bytes to each new SST file.

The `TablePropertiesNames` for these two identifiers are `rocksdb.creating.db.identity` and `rocksdb.creating.session.identity`.

In addition, SST files generated from SstFileWriter and Repairer have DB identity “SST Writer” and “DB Repairer”, respectively. Their DB session IDs are generated in the same way as `DB::GetDbSessionId`.

A table property test is added.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6983

Test Plan: make check and some manual tests.

Reviewed By: zhichao-cao

Differential Revision: D22048826

Pulled By: gg814

fbshipit-source-id: afdf8c11424a6f509b5c0b06dafad584a80103c9
main
Zitan Chen 4 years ago committed by Facebook GitHub Bot
parent 742b452863
commit 94d04529de
  1. 7
      HISTORY.md
  2. 10
      db/builder.cc
  3. 6
      db/builder.h
  4. 8
      db/compaction/compaction_job.cc
  5. 6
      db/compaction/compaction_job.h
  6. 11
      db/db_impl/db_impl_compaction_flush.cc
  7. 3
      db/db_impl/db_impl_open.cc
  8. 20
      db/db_table_properties_test.cc
  9. 4
      db/event_helpers.cc
  10. 8
      db/flush_job.cc
  11. 5
      db/flush_job.h
  12. 1
      db/flush_job_test.cc
  13. 7
      db/repair.cc
  14. 13
      include/rocksdb/table_properties.h
  15. 28
      table/block_based/block_based_table_builder.cc
  16. 3
      table/block_based/block_based_table_builder.h
  17. 3
      table/block_based/block_based_table_factory.cc
  18. 5
      table/cuckoo/cuckoo_table_builder.cc
  19. 15
      table/cuckoo/cuckoo_table_builder.h
  20. 3
      table/cuckoo/cuckoo_table_factory.cc
  21. 6
      table/meta_blocks.cc
  22. 5
      table/plain/plain_table_builder.cc
  23. 3
      table/plain/plain_table_builder.h
  24. 3
      table/plain/plain_table_factory.cc
  25. 14
      table/sst_file_writer.cc
  26. 10
      table/table_builder.h
  27. 8
      table/table_properties.cc

@ -4,6 +4,12 @@
* Best-efforts recovery ignores CURRENT file completely. If CURRENT file is missing during recovery, best-efforts recovery still proceeds with MANIFEST file(s). * Best-efforts recovery ignores CURRENT file completely. If CURRENT file is missing during recovery, best-efforts recovery still proceeds with MANIFEST file(s).
* In best-efforts recovery, an error that is not Corruption or IOError::kNotFound or IOError::kPathNotFound will be overwritten silently. Fix this by checking all non-ok cases and return early. * In best-efforts recovery, an error that is not Corruption or IOError::kNotFound or IOError::kPathNotFound will be overwritten silently. Fix this by checking all non-ok cases and return early.
### Public API Change
* `DB::GetDbSessionId(std::string& session_id)` is added. `session_id` stores a unique identifier that gets reset every time the DB is opened. This DB session ID should be unique among all open DB instances on all hosts, and should be unique among re-openings of the same or other DBs. This identifier is recorded in the LOG file on the line starting with "DB Session ID:".
### New Features
* DB identity (`db_id`) and DB session identity (`db_session_id`) are added to table properties and stored in SST files. SST files generated from SstFileWriter and Repairer have DB identity “SST Writer” and “DB Repairer”, respectively. Their DB session IDs are generated in the same way as `DB::GetDbSessionId`. The session ID for SstFileWriter (resp., Repairer) resets every time `SstFileWriter::Open` (resp., `Repairer::Run`) is called.
## 6.11 (6/12/2020) ## 6.11 (6/12/2020)
### Bug Fixes ### Bug Fixes
* Fix consistency checking error swallowing in some cases when options.force_consistency_checks = true. * Fix consistency checking error swallowing in some cases when options.force_consistency_checks = true.
@ -30,7 +36,6 @@
* `pin_l0_filter_and_index_blocks_in_cache` no longer applies to L0 files larger than `1.5 * write_buffer_size` to give more predictable memory usage. Such L0 files may exist due to intra-L0 compaction, external file ingestion, or user dynamically changing `write_buffer_size` (note, however, that files that are already pinned will continue being pinned, even after such a dynamic change). * `pin_l0_filter_and_index_blocks_in_cache` no longer applies to L0 files larger than `1.5 * write_buffer_size` to give more predictable memory usage. Such L0 files may exist due to intra-L0 compaction, external file ingestion, or user dynamically changing `write_buffer_size` (note, however, that files that are already pinned will continue being pinned, even after such a dynamic change).
* In point-in-time wal recovery mode, fail database recovery in case of IOError while reading the WAL to avoid data loss. * In point-in-time wal recovery mode, fail database recovery in case of IOError while reading the WAL to avoid data loss.
* A new method `Env::LowerThreadPoolCPUPriority(Priority, CpuPriority)` is added to `Env` to be able to lower to a specific priority such as `CpuPriority::kIdle`. * A new method `Env::LowerThreadPoolCPUPriority(Priority, CpuPriority)` is added to `Env` to be able to lower to a specific priority such as `CpuPriority::kIdle`.
* `DB::GetDbSessionId(std::string& session_id)` is added. `session_id` stores a unique identifier that gets reset every time the DB is opened. This DB session ID should be unique among all open DB instances on all hosts, and should be unique among re-openings of the same or other DBs. This identifier is recorded in the `LOG` file on the line starting with `DB Session ID:`.
### New Features ### New Features
* sst_dump to add a new --readahead_size argument. Users can specify read size when scanning the data. Sst_dump also tries to prefetch tail part of the SST files so usually some number of I/Os are saved there too. * sst_dump to add a new --readahead_size argument. Users can specify read size when scanning the data. Sst_dump also tries to prefetch tail part of the SST files so usually some number of I/Os are saved there too.

@ -51,7 +51,8 @@ TableBuilder* NewTableBuilder(
uint64_t sample_for_compression, const CompressionOptions& compression_opts, uint64_t sample_for_compression, const CompressionOptions& compression_opts,
int level, const bool skip_filters, const uint64_t creation_time, int level, const bool skip_filters, const uint64_t creation_time,
const uint64_t oldest_key_time, const uint64_t target_file_size, const uint64_t oldest_key_time, const uint64_t target_file_size,
const uint64_t file_creation_time) { const uint64_t file_creation_time, const std::string& db_id,
const std::string& db_session_id) {
assert((column_family_id == assert((column_family_id ==
TablePropertiesCollectorFactory::Context::kUnknownColumnFamily) == TablePropertiesCollectorFactory::Context::kUnknownColumnFamily) ==
column_family_name.empty()); column_family_name.empty());
@ -61,7 +62,7 @@ TableBuilder* NewTableBuilder(
sample_for_compression, compression_opts, sample_for_compression, compression_opts,
skip_filters, column_family_name, level, skip_filters, column_family_name, level,
creation_time, oldest_key_time, target_file_size, creation_time, oldest_key_time, target_file_size,
file_creation_time), file_creation_time, db_id, db_session_id),
column_family_id, file); column_family_id, file);
} }
@ -85,7 +86,8 @@ Status BuildTable(
EventLogger* event_logger, int job_id, const Env::IOPriority io_priority, EventLogger* event_logger, int job_id, const Env::IOPriority io_priority,
TableProperties* table_properties, int level, const uint64_t creation_time, TableProperties* table_properties, int level, const uint64_t creation_time,
const uint64_t oldest_key_time, Env::WriteLifeTimeHint write_hint, const uint64_t oldest_key_time, Env::WriteLifeTimeHint write_hint,
const uint64_t file_creation_time) { const uint64_t file_creation_time, const std::string& db_id,
const std::string& db_session_id) {
assert((column_family_id == assert((column_family_id ==
TablePropertiesCollectorFactory::Context::kUnknownColumnFamily) == TablePropertiesCollectorFactory::Context::kUnknownColumnFamily) ==
column_family_name.empty()); column_family_name.empty());
@ -142,7 +144,7 @@ Status BuildTable(
column_family_name, file_writer.get(), compression, column_family_name, file_writer.get(), compression,
sample_for_compression, compression_opts_for_flush, level, sample_for_compression, compression_opts_for_flush, level,
false /* skip_filters */, creation_time, oldest_key_time, false /* skip_filters */, creation_time, oldest_key_time,
0 /*target_file_size*/, file_creation_time); 0 /*target_file_size*/, file_creation_time, db_id, db_session_id);
} }
MergeHelper merge(env, internal_comparator.user_comparator(), MergeHelper merge(env, internal_comparator.user_comparator(),

@ -51,7 +51,8 @@ TableBuilder* NewTableBuilder(
const CompressionOptions& compression_opts, int level, const CompressionOptions& compression_opts, int level,
const bool skip_filters = false, const uint64_t creation_time = 0, const bool skip_filters = false, const uint64_t creation_time = 0,
const uint64_t oldest_key_time = 0, const uint64_t target_file_size = 0, const uint64_t oldest_key_time = 0, const uint64_t target_file_size = 0,
const uint64_t file_creation_time = 0); const uint64_t file_creation_time = 0, const std::string& db_id = "",
const std::string& db_session_id = "");
// Build a Table file from the contents of *iter. The generated file // Build a Table file from the contents of *iter. The generated file
// will be named according to number specified in meta. On success, the rest of // will be named according to number specified in meta. On success, the rest of
@ -83,6 +84,7 @@ extern Status BuildTable(
TableProperties* table_properties = nullptr, int level = -1, TableProperties* table_properties = nullptr, int level = -1,
const uint64_t creation_time = 0, const uint64_t oldest_key_time = 0, const uint64_t creation_time = 0, const uint64_t oldest_key_time = 0,
Env::WriteLifeTimeHint write_hint = Env::WLTH_NOT_SET, Env::WriteLifeTimeHint write_hint = Env::WLTH_NOT_SET,
const uint64_t file_creation_time = 0); const uint64_t file_creation_time = 0, const std::string& db_id = "",
const std::string& db_session_id = "");
} // namespace ROCKSDB_NAMESPACE } // namespace ROCKSDB_NAMESPACE

@ -305,12 +305,15 @@ CompactionJob::CompactionJob(
const SnapshotChecker* snapshot_checker, std::shared_ptr<Cache> table_cache, const SnapshotChecker* snapshot_checker, std::shared_ptr<Cache> table_cache,
EventLogger* event_logger, bool paranoid_file_checks, bool measure_io_stats, EventLogger* event_logger, bool paranoid_file_checks, bool measure_io_stats,
const std::string& dbname, CompactionJobStats* compaction_job_stats, const std::string& dbname, CompactionJobStats* compaction_job_stats,
Env::Priority thread_pri, const std::atomic<bool>* manual_compaction_paused) Env::Priority thread_pri, const std::atomic<bool>* manual_compaction_paused,
const std::string& db_id, const std::string& db_session_id)
: job_id_(job_id), : job_id_(job_id),
compact_(new CompactionState(compaction)), compact_(new CompactionState(compaction)),
compaction_job_stats_(compaction_job_stats), compaction_job_stats_(compaction_job_stats),
compaction_stats_(compaction->compaction_reason(), 1), compaction_stats_(compaction->compaction_reason(), 1),
dbname_(dbname), dbname_(dbname),
db_id_(db_id),
db_session_id_(db_session_id),
db_options_(db_options), db_options_(db_options),
file_options_(file_options), file_options_(file_options),
env_(db_options.env), env_(db_options.env),
@ -1554,7 +1557,8 @@ Status CompactionJob::OpenCompactionOutputFile(
sub_compact->compaction->output_compression_opts(), sub_compact->compaction->output_compression_opts(),
sub_compact->compaction->output_level(), skip_filters, sub_compact->compaction->output_level(), skip_filters,
oldest_ancester_time, 0 /* oldest_key_time */, oldest_ancester_time, 0 /* oldest_key_time */,
sub_compact->compaction->max_output_file_size(), current_time)); sub_compact->compaction->max_output_file_size(), current_time, db_id_,
db_session_id_));
LogFlush(db_options_.info_log); LogFlush(db_options_.info_log);
return s; return s;
} }

@ -78,7 +78,9 @@ class CompactionJob {
const std::string& dbname, const std::string& dbname,
CompactionJobStats* compaction_job_stats, CompactionJobStats* compaction_job_stats,
Env::Priority thread_pri, Env::Priority thread_pri,
const std::atomic<bool>* manual_compaction_paused = nullptr); const std::atomic<bool>* manual_compaction_paused = nullptr,
const std::string& db_id = "",
const std::string& db_session_id = "");
~CompactionJob(); ~CompactionJob();
@ -152,6 +154,8 @@ class CompactionJob {
// DBImpl state // DBImpl state
const std::string& dbname_; const std::string& dbname_;
const std::string db_id_;
const std::string db_session_id_;
const ImmutableDBOptions& db_options_; const ImmutableDBOptions& db_options_;
const FileOptions file_options_; const FileOptions file_options_;

@ -154,7 +154,8 @@ Status DBImpl::FlushMemTableToOutputFile(
GetDataDir(cfd, 0U), GetDataDir(cfd, 0U),
GetCompressionFlush(*cfd->ioptions(), mutable_cf_options), stats_, GetCompressionFlush(*cfd->ioptions(), mutable_cf_options), stats_,
&event_logger_, mutable_cf_options.report_bg_io_stats, &event_logger_, mutable_cf_options.report_bg_io_stats,
true /* sync_output_directory */, true /* write_manifest */, thread_pri); true /* sync_output_directory */, true /* write_manifest */, thread_pri,
db_id_, db_session_id_);
FileMetaData file_meta; FileMetaData file_meta;
TEST_SYNC_POINT("DBImpl::FlushMemTableToOutputFile:BeforePickMemtables"); TEST_SYNC_POINT("DBImpl::FlushMemTableToOutputFile:BeforePickMemtables");
@ -345,7 +346,7 @@ Status DBImpl::AtomicFlushMemTablesToOutputFiles(
data_dir, GetCompressionFlush(*cfd->ioptions(), mutable_cf_options), data_dir, GetCompressionFlush(*cfd->ioptions(), mutable_cf_options),
stats_, &event_logger_, mutable_cf_options.report_bg_io_stats, stats_, &event_logger_, mutable_cf_options.report_bg_io_stats,
false /* sync_output_directory */, false /* write_manifest */, false /* sync_output_directory */, false /* write_manifest */,
thread_pri)); thread_pri, db_id_, db_session_id_));
jobs.back()->PickMemTable(); jobs.back()->PickMemTable();
} }
@ -1031,7 +1032,8 @@ Status DBImpl::CompactFilesImpl(
snapshot_checker, table_cache_, &event_logger_, snapshot_checker, table_cache_, &event_logger_,
c->mutable_cf_options()->paranoid_file_checks, c->mutable_cf_options()->paranoid_file_checks,
c->mutable_cf_options()->report_bg_io_stats, dbname_, c->mutable_cf_options()->report_bg_io_stats, dbname_,
&compaction_job_stats, Env::Priority::USER, &manual_compaction_paused_); &compaction_job_stats, Env::Priority::USER, &manual_compaction_paused_,
db_id_, db_session_id_);
// Creating a compaction influences the compaction score because the score // Creating a compaction influences the compaction score because the score
// takes running compactions into account (by skipping files that are already // takes running compactions into account (by skipping files that are already
@ -2822,7 +2824,8 @@ Status DBImpl::BackgroundCompaction(bool* made_progress,
&event_logger_, c->mutable_cf_options()->paranoid_file_checks, &event_logger_, c->mutable_cf_options()->paranoid_file_checks,
c->mutable_cf_options()->report_bg_io_stats, dbname_, c->mutable_cf_options()->report_bg_io_stats, dbname_,
&compaction_job_stats, thread_pri, &compaction_job_stats, thread_pri,
is_manual ? &manual_compaction_paused_ : nullptr); is_manual ? &manual_compaction_paused_ : nullptr, db_id_,
db_session_id_);
compaction_job.Prepare(); compaction_job.Prepare();
NotifyOnCompactionBegin(c->column_family_data(), c.get(), status, NotifyOnCompactionBegin(c->column_family_data(), c.get(), status,

@ -1289,7 +1289,8 @@ Status DBImpl::WriteLevel0TableForRecovery(int job_id, ColumnFamilyData* cfd,
mutable_cf_options.compression_opts, paranoid_file_checks, mutable_cf_options.compression_opts, paranoid_file_checks,
cfd->internal_stats(), TableFileCreationReason::kRecovery, &io_s, cfd->internal_stats(), TableFileCreationReason::kRecovery, &io_s,
&event_logger_, job_id, Env::IO_HIGH, nullptr /* table_properties */, &event_logger_, job_id, Env::IO_HIGH, nullptr /* table_properties */,
-1 /* level */, current_time, write_hint); -1 /* level */, current_time, 0 /* oldest_key_time */, write_hint,
0 /* file_creation_time */, db_id_, db_session_id_);
LogFlush(immutable_db_options_.info_log); LogFlush(immutable_db_options_.info_log);
ROCKS_LOG_DEBUG(immutable_db_options_.info_log, ROCKS_LOG_DEBUG(immutable_db_options_.info_log,
"[%s] [WriteLevel0TableForRecovery]" "[%s] [WriteLevel0TableForRecovery]"

@ -252,6 +252,26 @@ TEST_F(DBTablePropertiesTest, GetColumnFamilyNameProperty) {
} }
} }
TEST_F(DBTablePropertiesTest, GetDbIdentifiersProperty) {
CreateAndReopenWithCF({"goku"}, CurrentOptions());
for (uint32_t cf = 0; cf < 2; ++cf) {
Put(cf, "key", "val");
Put(cf, "foo", "bar");
Flush(cf);
TablePropertiesCollection fname_to_props;
ASSERT_OK(db_->GetPropertiesOfAllTables(handles_[cf], &fname_to_props));
ASSERT_EQ(1U, fname_to_props.size());
std::string id, sid;
db_->GetDbIdentity(id);
db_->GetDbSessionId(sid);
ASSERT_EQ(id, fname_to_props.begin()->second->db_id);
ASSERT_EQ(sid, fname_to_props.begin()->second->db_session_id);
}
}
class DeletionTriggeredCompactionTestListener : public EventListener { class DeletionTriggeredCompactionTestListener : public EventListener {
public: public:
void OnCompactionBegin(DB* , const CompactionJobInfo& ci) override { void OnCompactionBegin(DB* , const CompactionJobInfo& ci) override {

@ -121,7 +121,9 @@ void EventHelpers::LogAndNotifyTableFileCreationFinished(
<< table_properties.compression_options << "creation_time" << table_properties.compression_options << "creation_time"
<< table_properties.creation_time << "oldest_key_time" << table_properties.creation_time << "oldest_key_time"
<< table_properties.oldest_key_time << "file_creation_time" << table_properties.oldest_key_time << "file_creation_time"
<< table_properties.file_creation_time; << table_properties.file_creation_time << "db_id"
<< table_properties.db_id << "db_session_id"
<< table_properties.db_session_id;
// user collected properties // user collected properties
for (const auto& prop : table_properties.readable_properties) { for (const auto& prop : table_properties.readable_properties) {

@ -97,8 +97,11 @@ FlushJob::FlushJob(const std::string& dbname, ColumnFamilyData* cfd,
CompressionType output_compression, Statistics* stats, CompressionType output_compression, Statistics* stats,
EventLogger* event_logger, bool measure_io_stats, EventLogger* event_logger, bool measure_io_stats,
const bool sync_output_directory, const bool write_manifest, const bool sync_output_directory, const bool write_manifest,
Env::Priority thread_pri) Env::Priority thread_pri, const std::string& db_id,
const std::string& db_session_id)
: dbname_(dbname), : dbname_(dbname),
db_id_(db_id),
db_session_id_(db_session_id),
cfd_(cfd), cfd_(cfd),
db_options_(db_options), db_options_(db_options),
mutable_cf_options_(mutable_cf_options), mutable_cf_options_(mutable_cf_options),
@ -393,7 +396,8 @@ Status FlushJob::WriteLevel0Table() {
mutable_cf_options_.paranoid_file_checks, cfd_->internal_stats(), mutable_cf_options_.paranoid_file_checks, cfd_->internal_stats(),
TableFileCreationReason::kFlush, &io_s, event_logger_, TableFileCreationReason::kFlush, &io_s, event_logger_,
job_context_->job_id, Env::IO_HIGH, &table_properties_, 0 /* level */, job_context_->job_id, Env::IO_HIGH, &table_properties_, 0 /* level */,
creation_time, oldest_key_time, write_hint, current_time); creation_time, oldest_key_time, write_hint, current_time, db_id_,
db_session_id_);
if (!io_s.ok()) { if (!io_s.ok()) {
io_status_ = io_s; io_status_ = io_s;
} }

@ -72,7 +72,8 @@ class FlushJob {
CompressionType output_compression, Statistics* stats, CompressionType output_compression, Statistics* stats,
EventLogger* event_logger, bool measure_io_stats, EventLogger* event_logger, bool measure_io_stats,
const bool sync_output_directory, const bool write_manifest, const bool sync_output_directory, const bool write_manifest,
Env::Priority thread_pri); Env::Priority thread_pri, const std::string& db_id = "",
const std::string& db_session_id = "");
~FlushJob(); ~FlushJob();
@ -103,6 +104,8 @@ class FlushJob {
#endif // !ROCKSDB_LITE #endif // !ROCKSDB_LITE
const std::string& dbname_; const std::string& dbname_;
const std::string db_id_;
const std::string db_session_id_;
ColumnFamilyData* cfd_; ColumnFamilyData* cfd_;
const ImmutableDBOptions& db_options_; const ImmutableDBOptions& db_options_;
const MutableCFOptions& mutable_cf_options_; const MutableCFOptions& mutable_cf_options_;

@ -282,7 +282,6 @@ TEST_F(FlushJobTest, FlushMemTablesSingleColumnFamily) {
assert(memtable_ids.size() == num_mems); assert(memtable_ids.size() == num_mems);
uint64_t smallest_memtable_id = memtable_ids.front(); uint64_t smallest_memtable_id = memtable_ids.front();
uint64_t flush_memtable_id = smallest_memtable_id + num_mems_to_flush - 1; uint64_t flush_memtable_id = smallest_memtable_id + num_mems_to_flush - 1;
FlushJob flush_job(dbname_, versions_->GetColumnFamilySet()->GetDefault(), FlushJob flush_job(dbname_, versions_->GetColumnFamilySet()->GetDefault(),
db_options_, *cfd->GetLatestMutableCFOptions(), db_options_, *cfd->GetLatestMutableCFOptions(),
&flush_memtable_id, env_options_, versions_.get(), &mutex_, &flush_memtable_id, env_options_, versions_.get(), &mutex_,

@ -183,6 +183,8 @@ class Repairer {
} }
// Just create a DBImpl temporarily so we can reuse NewDB() // Just create a DBImpl temporarily so we can reuse NewDB()
DBImpl* db_impl = new DBImpl(db_options_, dbname_); DBImpl* db_impl = new DBImpl(db_options_, dbname_);
// Also use this temp DBImpl to get a session id
db_impl->GetDbSessionId(db_session_id_);
status = db_impl->NewDB(); status = db_impl->NewDB();
delete db_impl; delete db_impl;
} }
@ -229,6 +231,7 @@ class Repairer {
}; };
std::string const dbname_; std::string const dbname_;
std::string db_session_id_;
Env* const env_; Env* const env_;
const EnvOptions env_options_; const EnvOptions env_options_;
const DBOptions db_options_; const DBOptions db_options_;
@ -435,7 +438,9 @@ class Repairer {
CompressionOptions(), false, nullptr /* internal_stats */, CompressionOptions(), false, nullptr /* internal_stats */,
TableFileCreationReason::kRecovery, &io_s, nullptr /* event_logger */, TableFileCreationReason::kRecovery, &io_s, nullptr /* event_logger */,
0 /* job_id */, Env::IO_HIGH, nullptr /* table_properties */, 0 /* job_id */, Env::IO_HIGH, nullptr /* table_properties */,
-1 /* level */, current_time, write_hint); -1 /* level */, current_time, 0 /* oldest_key_time */, write_hint,
0 /* file_creation_time */, "DB Repairer" /* db_id */,
db_session_id_);
ROCKS_LOG_INFO(db_options_.info_log, ROCKS_LOG_INFO(db_options_.info_log,
"Log #%" PRIu64 ": %d ops saved to Table #%" PRIu64 " %s", "Log #%" PRIu64 ": %d ops saved to Table #%" PRIu64 " %s",
log, counter, meta.fd.GetNumber(), log, counter, meta.fd.GetNumber(),

@ -30,6 +30,8 @@ typedef std::map<std::string, std::string> UserCollectedProperties;
// table properties' human-readable names in the property block. // table properties' human-readable names in the property block.
struct TablePropertiesNames { struct TablePropertiesNames {
static const std::string kDbId;
static const std::string kDbSessionId;
static const std::string kDataSize; static const std::string kDataSize;
static const std::string kIndexSize; static const std::string kIndexSize;
static const std::string kIndexPartitions; static const std::string kIndexPartitions;
@ -193,6 +195,17 @@ struct TableProperties {
// Actual SST file creation time. 0 means unknown. // Actual SST file creation time. 0 means unknown.
uint64_t file_creation_time = 0; uint64_t file_creation_time = 0;
// DB identity
// db_id is an identifier generated the first time the DB is created
// If DB identity is unset or unassigned, `db_id` will be an empty string.
std::string db_id;
// DB session identity
// db_session_id is an identifier that gets reset every time the DB is opened
// If DB session identity is unset or unassigned, `db_session_id` will be an
// empty string.
std::string db_session_id;
// Name of the column family with which this SST file is associated. // Name of the column family with which this SST file is associated.
// If column family is unknown, `column_family_name` will be an empty string. // If column family is unknown, `column_family_name` will be an empty string.
std::string column_family_name; std::string column_family_name;

@ -374,6 +374,10 @@ struct BlockBasedTableBuilder::Rep {
const uint64_t target_file_size; const uint64_t target_file_size;
uint64_t file_creation_time = 0; uint64_t file_creation_time = 0;
// DB IDs
const std::string db_id;
const std::string db_session_id;
std::vector<std::unique_ptr<IntTblPropCollector>> table_properties_collectors; std::vector<std::unique_ptr<IntTblPropCollector>> table_properties_collectors;
std::unique_ptr<ParallelCompressionRep> pc_rep; std::unique_ptr<ParallelCompressionRep> pc_rep;
@ -447,7 +451,8 @@ struct BlockBasedTableBuilder::Rep {
const CompressionOptions& _compression_opts, const bool skip_filters, const CompressionOptions& _compression_opts, const bool skip_filters,
const int _level_at_creation, const std::string& _column_family_name, const int _level_at_creation, const std::string& _column_family_name,
const uint64_t _creation_time, const uint64_t _oldest_key_time, const uint64_t _creation_time, const uint64_t _oldest_key_time,
const uint64_t _target_file_size, const uint64_t _file_creation_time) const uint64_t _target_file_size, const uint64_t _file_creation_time,
const std::string& _db_id, const std::string& _db_session_id)
: ioptions(_ioptions), : ioptions(_ioptions),
moptions(_moptions), moptions(_moptions),
table_options(table_opt), table_options(table_opt),
@ -488,7 +493,9 @@ struct BlockBasedTableBuilder::Rep {
creation_time(_creation_time), creation_time(_creation_time),
oldest_key_time(_oldest_key_time), oldest_key_time(_oldest_key_time),
target_file_size(_target_file_size), target_file_size(_target_file_size),
file_creation_time(_file_creation_time) { file_creation_time(_file_creation_time),
db_id(_db_id),
db_session_id(_db_session_id) {
for (uint32_t i = 0; i < compression_opts.parallel_threads; i++) { for (uint32_t i = 0; i < compression_opts.parallel_threads; i++) {
compression_ctxs[i].reset(new CompressionContext(compression_type)); compression_ctxs[i].reset(new CompressionContext(compression_type));
} }
@ -698,7 +705,8 @@ BlockBasedTableBuilder::BlockBasedTableBuilder(
const CompressionOptions& compression_opts, const bool skip_filters, const CompressionOptions& compression_opts, const bool skip_filters,
const std::string& column_family_name, const int level_at_creation, const std::string& column_family_name, const int level_at_creation,
const uint64_t creation_time, const uint64_t oldest_key_time, const uint64_t creation_time, const uint64_t oldest_key_time,
const uint64_t target_file_size, const uint64_t file_creation_time) { const uint64_t target_file_size, const uint64_t file_creation_time,
const std::string& db_id, const std::string& db_session_id) {
BlockBasedTableOptions sanitized_table_options(table_options); BlockBasedTableOptions sanitized_table_options(table_options);
if (sanitized_table_options.format_version == 0 && if (sanitized_table_options.format_version == 0 &&
sanitized_table_options.checksum != kCRC32c) { sanitized_table_options.checksum != kCRC32c) {
@ -711,12 +719,12 @@ BlockBasedTableBuilder::BlockBasedTableBuilder(
sanitized_table_options.format_version = 1; sanitized_table_options.format_version = 1;
} }
rep_ = new Rep(ioptions, moptions, sanitized_table_options, rep_ = new Rep(
internal_comparator, int_tbl_prop_collector_factories, ioptions, moptions, sanitized_table_options, internal_comparator,
column_family_id, file, compression_type, int_tbl_prop_collector_factories, column_family_id, file,
sample_for_compression, compression_opts, skip_filters, compression_type, sample_for_compression, compression_opts, skip_filters,
level_at_creation, column_family_name, creation_time, level_at_creation, column_family_name, creation_time, oldest_key_time,
oldest_key_time, target_file_size, file_creation_time); target_file_size, file_creation_time, db_id, db_session_id);
if (rep_->filter_builder != nullptr) { if (rep_->filter_builder != nullptr) {
rep_->filter_builder->StartBlock(0); rep_->filter_builder->StartBlock(0);
@ -1445,6 +1453,8 @@ void BlockBasedTableBuilder::WritePropertiesBlock(
rep_->props.creation_time = rep_->creation_time; rep_->props.creation_time = rep_->creation_time;
rep_->props.oldest_key_time = rep_->oldest_key_time; rep_->props.oldest_key_time = rep_->oldest_key_time;
rep_->props.file_creation_time = rep_->file_creation_time; rep_->props.file_creation_time = rep_->file_creation_time;
rep_->props.db_id = rep_->db_id;
rep_->props.db_session_id = rep_->db_session_id;
// Add basic properties // Add basic properties
property_block_builder.AddTableProperty(rep_->props); property_block_builder.AddTableProperty(rep_->props);

@ -51,7 +51,8 @@ class BlockBasedTableBuilder : public TableBuilder {
const std::string& column_family_name, const int level_at_creation, const std::string& column_family_name, const int level_at_creation,
const uint64_t creation_time = 0, const uint64_t oldest_key_time = 0, const uint64_t creation_time = 0, const uint64_t oldest_key_time = 0,
const uint64_t target_file_size = 0, const uint64_t target_file_size = 0,
const uint64_t file_creation_time = 0); const uint64_t file_creation_time = 0, const std::string& db_id = "",
const std::string& db_session_id = "");
// No copying allowed // No copying allowed
BlockBasedTableBuilder(const BlockBasedTableBuilder&) = delete; BlockBasedTableBuilder(const BlockBasedTableBuilder&) = delete;

@ -439,7 +439,8 @@ TableBuilder* BlockBasedTableFactory::NewTableBuilder(
table_builder_options.creation_time, table_builder_options.creation_time,
table_builder_options.oldest_key_time, table_builder_options.oldest_key_time,
table_builder_options.target_file_size, table_builder_options.target_file_size,
table_builder_options.file_creation_time); table_builder_options.file_creation_time, table_builder_options.db_id,
table_builder_options.db_session_id);
return table_builder; return table_builder;
} }

@ -53,7 +53,8 @@ CuckooTableBuilder::CuckooTableBuilder(
const Comparator* user_comparator, uint32_t cuckoo_block_size, const Comparator* user_comparator, uint32_t cuckoo_block_size,
bool use_module_hash, bool identity_as_first_hash, bool use_module_hash, bool identity_as_first_hash,
uint64_t (*get_slice_hash)(const Slice&, uint32_t, uint64_t), uint64_t (*get_slice_hash)(const Slice&, uint32_t, uint64_t),
uint32_t column_family_id, const std::string& column_family_name) uint32_t column_family_id, const std::string& column_family_name,
const std::string& db_id, const std::string& db_session_id)
: num_hash_func_(2), : num_hash_func_(2),
file_(file), file_(file),
max_hash_table_ratio_(max_hash_table_ratio), max_hash_table_ratio_(max_hash_table_ratio),
@ -79,6 +80,8 @@ CuckooTableBuilder::CuckooTableBuilder(
properties_.filter_size = 0; properties_.filter_size = 0;
properties_.column_family_id = column_family_id; properties_.column_family_id = column_family_id;
properties_.column_family_name = column_family_name; properties_.column_family_name = column_family_name;
properties_.db_id = db_id;
properties_.db_session_id = db_session_id;
} }
void CuckooTableBuilder::Add(const Slice& key, const Slice& value) { void CuckooTableBuilder::Add(const Slice& key, const Slice& value) {

@ -22,15 +22,14 @@ namespace ROCKSDB_NAMESPACE {
class CuckooTableBuilder: public TableBuilder { class CuckooTableBuilder: public TableBuilder {
public: public:
CuckooTableBuilder(WritableFileWriter* file, double max_hash_table_ratio, CuckooTableBuilder(
WritableFileWriter* file, double max_hash_table_ratio,
uint32_t max_num_hash_func, uint32_t max_search_depth, uint32_t max_num_hash_func, uint32_t max_search_depth,
const Comparator* user_comparator, const Comparator* user_comparator, uint32_t cuckoo_block_size,
uint32_t cuckoo_block_size, bool use_module_hash, bool use_module_hash, bool identity_as_first_hash,
bool identity_as_first_hash, uint64_t (*get_slice_hash)(const Slice&, uint32_t, uint64_t),
uint64_t (*get_slice_hash)(const Slice&, uint32_t, uint32_t column_family_id, const std::string& column_family_name,
uint64_t), const std::string& db_id = "", const std::string& db_session_id = "");
uint32_t column_family_id,
const std::string& column_family_name);
// No copying allowed // No copying allowed
CuckooTableBuilder(const CuckooTableBuilder&) = delete; CuckooTableBuilder(const CuckooTableBuilder&) = delete;
void operator=(const CuckooTableBuilder&) = delete; void operator=(const CuckooTableBuilder&) = delete;

@ -40,7 +40,8 @@ TableBuilder* CuckooTableFactory::NewTableBuilder(
table_builder_options.internal_comparator.user_comparator(), table_builder_options.internal_comparator.user_comparator(),
table_options_.cuckoo_block_size, table_options_.use_module_hash, table_options_.cuckoo_block_size, table_options_.use_module_hash,
table_options_.identity_as_first_hash, nullptr /* get_slice_hash */, table_options_.identity_as_first_hash, nullptr /* get_slice_hash */,
column_family_id, table_builder_options.column_family_name); column_family_id, table_builder_options.column_family_name,
table_builder_options.db_id, table_builder_options.db_session_id);
} }
std::string CuckooTableFactory::GetPrintableTableOptions() const { std::string CuckooTableFactory::GetPrintableTableOptions() const {

@ -96,6 +96,8 @@ void PropertyBlockBuilder::AddTableProperty(const TableProperties& props) {
if (props.file_creation_time > 0) { if (props.file_creation_time > 0) {
Add(TablePropertiesNames::kFileCreationTime, props.file_creation_time); Add(TablePropertiesNames::kFileCreationTime, props.file_creation_time);
} }
Add(TablePropertiesNames::kDbId, props.db_id);
Add(TablePropertiesNames::kDbSessionId, props.db_session_id);
if (!props.filter_policy_name.empty()) { if (!props.filter_policy_name.empty()) {
Add(TablePropertiesNames::kFilterPolicy, props.filter_policy_name); Add(TablePropertiesNames::kFilterPolicy, props.filter_policy_name);
@ -311,6 +313,10 @@ Status ReadProperties(const Slice& handle_value, RandomAccessFileReader* file,
continue; continue;
} }
*(pos->second) = val; *(pos->second) = val;
} else if (key == TablePropertiesNames::kDbId) {
new_table_properties->db_id = raw_val.ToString();
} else if (key == TablePropertiesNames::kDbSessionId) {
new_table_properties->db_session_id = raw_val.ToString();
} else if (key == TablePropertiesNames::kFilterPolicy) { } else if (key == TablePropertiesNames::kFilterPolicy) {
new_table_properties->filter_policy_name = raw_val.ToString(); new_table_properties->filter_policy_name = raw_val.ToString();
} else if (key == TablePropertiesNames::kColumnFamilyName) { } else if (key == TablePropertiesNames::kColumnFamilyName) {

@ -64,7 +64,8 @@ PlainTableBuilder::PlainTableBuilder(
EncodingType encoding_type, size_t index_sparseness, EncodingType encoding_type, size_t index_sparseness,
uint32_t bloom_bits_per_key, const std::string& column_family_name, uint32_t bloom_bits_per_key, const std::string& column_family_name,
uint32_t num_probes, size_t huge_page_tlb_size, double hash_table_ratio, uint32_t num_probes, size_t huge_page_tlb_size, double hash_table_ratio,
bool store_index_in_file) bool store_index_in_file, const std::string& db_id,
const std::string& db_session_id)
: ioptions_(ioptions), : ioptions_(ioptions),
moptions_(moptions), moptions_(moptions),
bloom_block_(num_probes), bloom_block_(num_probes),
@ -97,6 +98,8 @@ PlainTableBuilder::PlainTableBuilder(
properties_.format_version = (encoding_type == kPlain) ? 0 : 1; properties_.format_version = (encoding_type == kPlain) ? 0 : 1;
properties_.column_family_id = column_family_id; properties_.column_family_id = column_family_id;
properties_.column_family_name = column_family_name; properties_.column_family_name = column_family_name;
properties_.db_id = db_id;
properties_.db_session_id = db_session_id;
properties_.prefix_extractor_name = moptions_.prefix_extractor != nullptr properties_.prefix_extractor_name = moptions_.prefix_extractor != nullptr
? moptions_.prefix_extractor->Name() ? moptions_.prefix_extractor->Name()
: "nullptr"; : "nullptr";

@ -45,7 +45,8 @@ class PlainTableBuilder: public TableBuilder {
size_t index_sparseness, uint32_t bloom_bits_per_key, size_t index_sparseness, uint32_t bloom_bits_per_key,
const std::string& column_family_name, uint32_t num_probes = 6, const std::string& column_family_name, uint32_t num_probes = 6,
size_t huge_page_tlb_size = 0, double hash_table_ratio = 0, size_t huge_page_tlb_size = 0, double hash_table_ratio = 0,
bool store_index_in_file = false); bool store_index_in_file = false, const std::string& db_id = "",
const std::string& db_session_id = "");
// No copying allowed // No copying allowed
PlainTableBuilder(const PlainTableBuilder&) = delete; PlainTableBuilder(const PlainTableBuilder&) = delete;
void operator=(const PlainTableBuilder&) = delete; void operator=(const PlainTableBuilder&) = delete;

@ -76,7 +76,8 @@ TableBuilder* PlainTableFactory::NewTableBuilder(
table_options_.index_sparseness, table_options_.bloom_bits_per_key, table_options_.index_sparseness, table_options_.bloom_bits_per_key,
table_builder_options.column_family_name, 6, table_builder_options.column_family_name, 6,
table_options_.huge_page_tlb_size, table_options_.hash_table_ratio, table_options_.huge_page_tlb_size, table_options_.hash_table_ratio,
table_options_.store_index_in_file); table_options_.store_index_in_file, table_builder_options.db_id,
table_builder_options.db_session_id);
} }
std::string PlainTableFactory::GetPrintableTableOptions() const { std::string PlainTableFactory::GetPrintableTableOptions() const {

@ -237,12 +237,22 @@ Status SstFileWriter::Open(const std::string& file_path) {
r->column_family_name = ""; r->column_family_name = "";
cf_id = TablePropertiesCollectorFactory::Context::kUnknownColumnFamily; cf_id = TablePropertiesCollectorFactory::Context::kUnknownColumnFamily;
} }
// SstFileWriter is used to create sst files that can be added to database
// later. Therefore, no real db_id and db_session_id are associated with it.
// Here we mimic the way db_session_id behaves by resetting the db_session_id
// every time SstFileWriter is used, and in this case db_id is set to be "SST
// Writer".
std::string db_session_id = r->ioptions.env->GenerateUniqueId();
if (!db_session_id.empty() && db_session_id.back() == '\n') {
db_session_id.pop_back();
}
TableBuilderOptions table_builder_options( TableBuilderOptions table_builder_options(
r->ioptions, r->mutable_cf_options, r->internal_comparator, r->ioptions, r->mutable_cf_options, r->internal_comparator,
&int_tbl_prop_collector_factories, compression_type, &int_tbl_prop_collector_factories, compression_type,
sample_for_compression, compression_opts, r->skip_filters, sample_for_compression, compression_opts, r->skip_filters,
r->column_family_name, unknown_level); r->column_family_name, unknown_level, 0 /* creation_time */,
0 /* oldest_key_time */, 0 /* target_file_size */,
0 /* file_creation_time */, "SST Writer" /* db_id */, db_session_id);
r->file_writer.reset(new WritableFileWriter( r->file_writer.reset(new WritableFileWriter(
NewLegacyWritableFileWrapper(std::move(sst_file)), file_path, NewLegacyWritableFileWrapper(std::move(sst_file)), file_path,
r->env_options, r->ioptions.env, nullptr /* stats */, r->env_options, r->ioptions.env, nullptr /* stats */,

@ -97,7 +97,8 @@ struct TableBuilderOptions {
const std::string& _column_family_name, int _level, const std::string& _column_family_name, int _level,
const uint64_t _creation_time = 0, const int64_t _oldest_key_time = 0, const uint64_t _creation_time = 0, const int64_t _oldest_key_time = 0,
const uint64_t _target_file_size = 0, const uint64_t _target_file_size = 0,
const uint64_t _file_creation_time = 0) const uint64_t _file_creation_time = 0, const std::string& _db_id = "",
const std::string& _db_session_id = "")
: ioptions(_ioptions), : ioptions(_ioptions),
moptions(_moptions), moptions(_moptions),
internal_comparator(_internal_comparator), internal_comparator(_internal_comparator),
@ -111,7 +112,10 @@ struct TableBuilderOptions {
creation_time(_creation_time), creation_time(_creation_time),
oldest_key_time(_oldest_key_time), oldest_key_time(_oldest_key_time),
target_file_size(_target_file_size), target_file_size(_target_file_size),
file_creation_time(_file_creation_time) {} file_creation_time(_file_creation_time),
db_id(_db_id),
db_session_id(_db_session_id) {}
const ImmutableCFOptions& ioptions; const ImmutableCFOptions& ioptions;
const MutableCFOptions& moptions; const MutableCFOptions& moptions;
const InternalKeyComparator& internal_comparator; const InternalKeyComparator& internal_comparator;
@ -127,6 +131,8 @@ struct TableBuilderOptions {
const int64_t oldest_key_time; const int64_t oldest_key_time;
const uint64_t target_file_size; const uint64_t target_file_size;
const uint64_t file_creation_time; const uint64_t file_creation_time;
const std::string db_id;
const std::string db_session_id;
}; };
// TableBuilder provides the interface used to build a Table // TableBuilder provides the interface used to build a Table

@ -168,6 +168,11 @@ std::string TableProperties::ToString(
AppendProperty(result, "file creation time", file_creation_time, prop_delim, AppendProperty(result, "file creation time", file_creation_time, prop_delim,
kv_delim); kv_delim);
// DB identity and DB session ID
AppendProperty(result, "DB identity", db_id, prop_delim, kv_delim);
AppendProperty(result, "DB session identity", db_session_id, prop_delim,
kv_delim);
return result; return result;
} }
@ -188,6 +193,9 @@ void TableProperties::Add(const TableProperties& tp) {
num_range_deletions += tp.num_range_deletions; num_range_deletions += tp.num_range_deletions;
} }
const std::string TablePropertiesNames::kDbId = "rocksdb.creating.db.identity";
const std::string TablePropertiesNames::kDbSessionId =
"rocksdb.creating.session.identity";
const std::string TablePropertiesNames::kDataSize = const std::string TablePropertiesNames::kDataSize =
"rocksdb.data.size"; "rocksdb.data.size";
const std::string TablePropertiesNames::kIndexSize = const std::string TablePropertiesNames::kIndexSize =

Loading…
Cancel
Save