LOG more info on oldest snapshot and sequence numbers (#10454)

Summary:
The info LOG file does not currently give any direct
information about the existence of old, live snapshots, nor how to
estimate wall time from a sequence number within the scope of LOG
history. This change addresses both with:
* Logging smallest and largest seqnos for generated SST files, which can
help associate sequence numbers with write time (based on flushes).
* Logging oldest_snapshot_seqno for each compaction, which (along with
that seqno info) helps us to determine how much old data might be kept
around for old (leaked?) snapshots. Including the date here I thought might
be excessive.

I wanted to log the date and seqno of the oldest snapshot with periodic
stats, but the current structure of the code doesn't really support that
because `DumpDBStats` doesn't have access to the DB object.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/10454

Test Plan:
manual inspect LOG from
`KEEP_DB=1 ./db_basic_test --gtest_filter=*CompactBetweenSnapshots*`

Reviewed By: ajkr

Differential Revision: D38326948

Pulled By: pdillinger

fbshipit-source-id: 294918ffc04a419844146cd826045321b4d5c038
main
Peter Dillinger 2 years ago committed by Facebook GitHub Bot
parent 2297769b38
commit 9fa5c146d7
  1. 5
      db/compaction/compaction_job.cc
  2. 3
      db/event_helpers.cc

@ -1961,7 +1961,10 @@ void CompactionJob::LogCompaction() {
stream.EndArray(); stream.EndArray();
} }
stream << "score" << compaction->score() << "input_data_size" stream << "score" << compaction->score() << "input_data_size"
<< compaction->CalculateTotalInputSize(); << compaction->CalculateTotalInputSize() << "oldest_snapshot_seqno"
<< (existing_snapshots_.empty()
? int64_t{-1} // Use -1 for "none"
: static_cast<int64_t>(existing_snapshots_[0]));
} }
} }

@ -97,7 +97,8 @@ void EventHelpers::LogAndNotifyTableFileCreationFinished(
<< "file_number" << fd.GetNumber() << "file_size" << "file_number" << fd.GetNumber() << "file_size"
<< fd.GetFileSize() << "file_checksum" << fd.GetFileSize() << "file_checksum"
<< Slice(file_checksum).ToString(true) << "file_checksum_func_name" << Slice(file_checksum).ToString(true) << "file_checksum_func_name"
<< file_checksum_func_name; << file_checksum_func_name << "smallest_seqno" << fd.smallest_seqno
<< "largest_seqno" << fd.largest_seqno;
// table_properties // table_properties
{ {

Loading…
Cancel
Save