From 0cdaa1a804d69f24a8193bc48117af5cce3eb576 Mon Sep 17 00:00:00 2001 From: Andrew Kryczka Date: Wed, 14 Mar 2018 15:59:26 -0700 Subject: [PATCH] Fix WAL corruption from checkpoint/backup race condition Summary: `Writer::WriteBuffer` was always called at the beginning of checkpoint/backup. But that log writer has no internal synchronization, which meant the same buffer could be flushed twice in a race condition case, causing a WAL entry to be duplicated. Then subsequent WAL entries would be at unexpected offsets, causing the 32KB block boundaries to be overlapped and manifesting as a corruption. This PR fixes the behavior to only use `WriteBuffer` (via `FlushWAL`) in checkpoint/backup when manual WAL flush is enabled. In that case, users are responsible for providing synchronization between WAL flushes. We can also consider removing the call entirely. Closes https://github.com/facebook/rocksdb/pull/3603 Differential Revision: D7277447 Pulled By: ajkr fbshipit-source-id: 1b15bd7fd930511222b075418c10de0aaa70a35a --- HISTORY.md | 1 + utilities/checkpoint/checkpoint_impl.cc | 4 +++- 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/HISTORY.md b/HISTORY.md index 6c5b999e3..ce619b1bf 100644 --- a/HISTORY.md +++ b/HISTORY.md @@ -12,6 +12,7 @@ ### Bug Fixes * Fix a leak in prepared_section_completed_ where the zeroed entries would not removed from the map. +* Fix WAL corruption caused by race condition between user write thread and backup/checkpoint thread. ## 5.12.0 (2/14/2018) ### Public API Change diff --git a/utilities/checkpoint/checkpoint_impl.cc b/utilities/checkpoint/checkpoint_impl.cc index 9135dbabf..9b0568c22 100644 --- a/utilities/checkpoint/checkpoint_impl.cc +++ b/utilities/checkpoint/checkpoint_impl.cc @@ -222,7 +222,9 @@ Status CheckpointImpl::CreateCustomCheckpoint( TEST_SYNC_POINT("CheckpointImpl::CreateCheckpoint:SavedLiveFiles1"); TEST_SYNC_POINT("CheckpointImpl::CreateCheckpoint:SavedLiveFiles2"); - db_->FlushWAL(false /* sync */); + if (db_options.manual_wal_flush) { + db_->FlushWAL(false /* sync */); + } } // if we have more than one column family, we need to also get WAL files if (s.ok()) {