|
|
|
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
|
|
|
|
// This source code is licensed under both the GPLv2 (found in the
|
|
|
|
// COPYING file in the root directory) and Apache 2.0 License
|
|
|
|
// (found in the LICENSE.Apache file in the root directory).
|
|
|
|
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
|
|
|
|
// Use of this source code is governed by a BSD-style license that can be
|
|
|
|
// found in the LICENSE file. See the AUTHORS file for names of contributors.
|
|
|
|
//
|
|
|
|
// An iterator yields a sequence of key/value pairs from a source.
|
|
|
|
// The following class defines the interface. Multiple implementations
|
|
|
|
// are provided by this library. In particular, iterators are provided
|
|
|
|
// to access the contents of a Table or a DB.
|
|
|
|
//
|
|
|
|
// Multiple threads can invoke const methods on an Iterator without
|
|
|
|
// external synchronization, but if any of the threads may call a
|
|
|
|
// non-const method, all threads accessing the same Iterator must use
|
|
|
|
// external synchronization.
|
|
|
|
|
|
|
|
#pragma once
|
|
|
|
|
|
|
|
#include <string>
|
|
|
|
|
|
|
|
#include "rocksdb/cleanable.h"
|
|
|
|
#include "rocksdb/slice.h"
|
|
|
|
#include "rocksdb/status.h"
|
|
|
|
#include "rocksdb/wide_columns.h"
|
|
|
|
|
|
|
|
namespace ROCKSDB_NAMESPACE {
|
|
|
|
|
|
|
|
class Iterator : public Cleanable {
|
|
|
|
public:
|
|
|
|
Iterator() {}
|
|
|
|
// No copying allowed
|
|
|
|
Iterator(const Iterator&) = delete;
|
|
|
|
void operator=(const Iterator&) = delete;
|
|
|
|
|
|
|
|
virtual ~Iterator() {}
|
|
|
|
|
|
|
|
// An iterator is either positioned at a key/value pair, or
|
|
|
|
// not valid. This method returns true iff the iterator is valid.
|
Change and clarify the relationship between Valid(), status() and Seek*() for all iterators. Also fix some bugs
Summary:
Before this PR, Iterator/InternalIterator may simultaneously have non-ok status() and Valid() = true. That state means that the last operation failed, but the iterator is nevertheless positioned on some unspecified record. Likely intended uses of that are:
* If some sst files are corrupted, a normal iterator can be used to read the data from files that are not corrupted.
* When using read_tier = kBlockCacheTier, read the data that's in block cache, skipping over the data that is not.
However, this behavior wasn't documented well (and until recently the wiki on github had misleading incorrect information). In the code there's a lot of confusion about the relationship between status() and Valid(), and about whether Seek()/SeekToLast()/etc reset the status or not. There were a number of bugs caused by this confusion, both inside rocksdb and in the code that uses rocksdb (including ours).
This PR changes the convention to:
* If status() is not ok, Valid() always returns false.
* Any seek operation resets status. (Before the PR, it depended on iterator type and on particular error.)
This does sacrifice the two use cases listed above, but siying said it's ok.
Overview of the changes:
* A commit that adds missing status checks in MergingIterator. This fixes a bug that actually affects us, and we need it fixed. `DBIteratorTest.NonBlockingIterationBugRepro` explains the scenario.
* Changes to lots of iterator types to make all of them conform to the new convention. Some bug fixes along the way. By far the biggest changes are in DBIter, which is a big messy piece of code; I tried to make it less big and messy but mostly failed.
* A stress-test for DBIter, to gain some confidence that I didn't break it. It does a few million random operations on the iterator, while occasionally modifying the underlying data (like ForwardIterator does) and occasionally returning non-ok status from internal iterator.
To find the iterator types that needed changes I searched for "public .*Iterator" in the code. Here's an overview of all 27 iterator types:
Iterators that didn't need changes:
* status() is always ok(), or Valid() is always false: MemTableIterator, ModelIter, TestIterator, KVIter (2 classes with this name anonymous namespaces), LoggingForwardVectorIterator, VectorIterator, MockTableIterator, EmptyIterator, EmptyInternalIterator.
* Thin wrappers that always pass through Valid() and status(): ArenaWrappedDBIter, TtlIterator, InternalIteratorFromIterator.
Iterators with changes (see inline comments for details):
* DBIter - an overhaul:
- It used to silently skip corrupted keys (`FindParseableKey()`), which seems dangerous. This PR makes it just stop immediately after encountering a corrupted key, just like it would for other kinds of corruption. Let me know if there was actually some deeper meaning in this behavior and I should put it back.
- It had a few code paths silently discarding subiterator's status. The stress test caught a few.
- The backwards iteration code path was expecting the internal iterator's set of keys to be immutable. It's probably always true in practice at the moment, since ForwardIterator doesn't support backwards iteration, but this PR fixes it anyway. See added DBIteratorTest.ReverseToForwardBug for an example.
- Some parts of backwards iteration code path even did things like `assert(iter_->Valid())` after a seek, which is never a safe assumption.
- It used to not reset status on seek for some types of errors.
- Some simplifications and better comments.
- Some things got more complicated from the added error handling. I'm open to ideas for how to make it nicer.
* MergingIterator - check status after every operation on every subiterator, and in some places assert that valid subiterators have ok status.
* ForwardIterator - changed to the new convention, also slightly simplified.
* ForwardLevelIterator - fixed some bugs and simplified.
* LevelIterator - simplified.
* TwoLevelIterator - changed to the new convention. Also fixed a bug that would make SeekForPrev() sometimes silently ignore errors from first_level_iter_.
* BlockBasedTableIterator - minor changes.
* BlockIter - replaced `SetStatus()` with `Invalidate()` to make sure non-ok BlockIter is always invalid.
* PlainTableIterator - some seeks used to not reset status.
* CuckooTableIterator - tiny code cleanup.
* ManagedIterator - fixed some bugs.
* BaseDeltaIterator - changed to the new convention and fixed a bug.
* BlobDBIterator - seeks used to not reset status.
* KeyConvertingIterator - some small change.
Closes https://github.com/facebook/rocksdb/pull/3810
Differential Revision: D7888019
Pulled By: al13n321
fbshipit-source-id: 4aaf6d3421c545d16722a815b2fa2e7912bc851d
7 years ago
|
|
|
// Always returns false if !status().ok().
|
|
|
|
virtual bool Valid() const = 0;
|
|
|
|
|
|
|
|
// Position at the first key in the source. The iterator is Valid()
|
|
|
|
// after this call iff the source is not empty.
|
|
|
|
virtual void SeekToFirst() = 0;
|
|
|
|
|
|
|
|
// Position at the last key in the source. The iterator is
|
|
|
|
// Valid() after this call iff the source is not empty.
|
|
|
|
virtual void SeekToLast() = 0;
|
|
|
|
|
|
|
|
// Position at the first key in the source that at or past target.
|
|
|
|
// The iterator is Valid() after this call iff the source contains
|
|
|
|
// an entry that comes at or past target.
|
Change and clarify the relationship between Valid(), status() and Seek*() for all iterators. Also fix some bugs
Summary:
Before this PR, Iterator/InternalIterator may simultaneously have non-ok status() and Valid() = true. That state means that the last operation failed, but the iterator is nevertheless positioned on some unspecified record. Likely intended uses of that are:
* If some sst files are corrupted, a normal iterator can be used to read the data from files that are not corrupted.
* When using read_tier = kBlockCacheTier, read the data that's in block cache, skipping over the data that is not.
However, this behavior wasn't documented well (and until recently the wiki on github had misleading incorrect information). In the code there's a lot of confusion about the relationship between status() and Valid(), and about whether Seek()/SeekToLast()/etc reset the status or not. There were a number of bugs caused by this confusion, both inside rocksdb and in the code that uses rocksdb (including ours).
This PR changes the convention to:
* If status() is not ok, Valid() always returns false.
* Any seek operation resets status. (Before the PR, it depended on iterator type and on particular error.)
This does sacrifice the two use cases listed above, but siying said it's ok.
Overview of the changes:
* A commit that adds missing status checks in MergingIterator. This fixes a bug that actually affects us, and we need it fixed. `DBIteratorTest.NonBlockingIterationBugRepro` explains the scenario.
* Changes to lots of iterator types to make all of them conform to the new convention. Some bug fixes along the way. By far the biggest changes are in DBIter, which is a big messy piece of code; I tried to make it less big and messy but mostly failed.
* A stress-test for DBIter, to gain some confidence that I didn't break it. It does a few million random operations on the iterator, while occasionally modifying the underlying data (like ForwardIterator does) and occasionally returning non-ok status from internal iterator.
To find the iterator types that needed changes I searched for "public .*Iterator" in the code. Here's an overview of all 27 iterator types:
Iterators that didn't need changes:
* status() is always ok(), or Valid() is always false: MemTableIterator, ModelIter, TestIterator, KVIter (2 classes with this name anonymous namespaces), LoggingForwardVectorIterator, VectorIterator, MockTableIterator, EmptyIterator, EmptyInternalIterator.
* Thin wrappers that always pass through Valid() and status(): ArenaWrappedDBIter, TtlIterator, InternalIteratorFromIterator.
Iterators with changes (see inline comments for details):
* DBIter - an overhaul:
- It used to silently skip corrupted keys (`FindParseableKey()`), which seems dangerous. This PR makes it just stop immediately after encountering a corrupted key, just like it would for other kinds of corruption. Let me know if there was actually some deeper meaning in this behavior and I should put it back.
- It had a few code paths silently discarding subiterator's status. The stress test caught a few.
- The backwards iteration code path was expecting the internal iterator's set of keys to be immutable. It's probably always true in practice at the moment, since ForwardIterator doesn't support backwards iteration, but this PR fixes it anyway. See added DBIteratorTest.ReverseToForwardBug for an example.
- Some parts of backwards iteration code path even did things like `assert(iter_->Valid())` after a seek, which is never a safe assumption.
- It used to not reset status on seek for some types of errors.
- Some simplifications and better comments.
- Some things got more complicated from the added error handling. I'm open to ideas for how to make it nicer.
* MergingIterator - check status after every operation on every subiterator, and in some places assert that valid subiterators have ok status.
* ForwardIterator - changed to the new convention, also slightly simplified.
* ForwardLevelIterator - fixed some bugs and simplified.
* LevelIterator - simplified.
* TwoLevelIterator - changed to the new convention. Also fixed a bug that would make SeekForPrev() sometimes silently ignore errors from first_level_iter_.
* BlockBasedTableIterator - minor changes.
* BlockIter - replaced `SetStatus()` with `Invalidate()` to make sure non-ok BlockIter is always invalid.
* PlainTableIterator - some seeks used to not reset status.
* CuckooTableIterator - tiny code cleanup.
* ManagedIterator - fixed some bugs.
* BaseDeltaIterator - changed to the new convention and fixed a bug.
* BlobDBIterator - seeks used to not reset status.
* KeyConvertingIterator - some small change.
Closes https://github.com/facebook/rocksdb/pull/3810
Differential Revision: D7888019
Pulled By: al13n321
fbshipit-source-id: 4aaf6d3421c545d16722a815b2fa2e7912bc851d
7 years ago
|
|
|
// All Seek*() methods clear any error status() that the iterator had prior to
|
|
|
|
// the call; after the seek, status() indicates only the error (if any) that
|
|
|
|
// happened during the seek, not any past errors.
|
|
|
|
// Target does not contain timestamp.
|
|
|
|
virtual void Seek(const Slice& target) = 0;
|
|
|
|
|
|
|
|
// Position at the last key in the source that at or before target.
|
|
|
|
// The iterator is Valid() after this call iff the source contains
|
|
|
|
// an entry that comes at or before target.
|
|
|
|
// Target does not contain timestamp.
|
|
|
|
virtual void SeekForPrev(const Slice& target) = 0;
|
|
|
|
|
|
|
|
// Moves to the next entry in the source. After this call, Valid() is
|
|
|
|
// true iff the iterator was not positioned at the last entry in the source.
|
|
|
|
// REQUIRES: Valid()
|
|
|
|
virtual void Next() = 0;
|
|
|
|
|
|
|
|
// Moves to the previous entry in the source. After this call, Valid() is
|
|
|
|
// true iff the iterator was not positioned at the first entry in source.
|
|
|
|
// REQUIRES: Valid()
|
|
|
|
virtual void Prev() = 0;
|
|
|
|
|
|
|
|
// Return the key for the current entry. The underlying storage for
|
|
|
|
// the returned slice is valid only until the next modification of the
|
|
|
|
// iterator (i.e. the next SeekToFirst/SeekToLast/Seek/SeekForPrev/Next/Prev
|
|
|
|
// operation).
|
|
|
|
// REQUIRES: Valid()
|
|
|
|
virtual Slice key() const = 0;
|
|
|
|
|
|
|
|
// Return the value for the current entry. If the entry is a plain key-value,
|
|
|
|
// return the value as-is; if it is a wide-column entity, return the value of
|
|
|
|
// the default anonymous column (see kDefaultWideColumnName) if any, or an
|
|
|
|
// empty value otherwise. The underlying storage for the returned slice is
|
|
|
|
// valid only until the next modification of the iterator (i.e. the next
|
|
|
|
// SeekToFirst/SeekToLast/Seek/SeekForPrev/Next/Prev operation).
|
Change and clarify the relationship between Valid(), status() and Seek*() for all iterators. Also fix some bugs
Summary:
Before this PR, Iterator/InternalIterator may simultaneously have non-ok status() and Valid() = true. That state means that the last operation failed, but the iterator is nevertheless positioned on some unspecified record. Likely intended uses of that are:
* If some sst files are corrupted, a normal iterator can be used to read the data from files that are not corrupted.
* When using read_tier = kBlockCacheTier, read the data that's in block cache, skipping over the data that is not.
However, this behavior wasn't documented well (and until recently the wiki on github had misleading incorrect information). In the code there's a lot of confusion about the relationship between status() and Valid(), and about whether Seek()/SeekToLast()/etc reset the status or not. There were a number of bugs caused by this confusion, both inside rocksdb and in the code that uses rocksdb (including ours).
This PR changes the convention to:
* If status() is not ok, Valid() always returns false.
* Any seek operation resets status. (Before the PR, it depended on iterator type and on particular error.)
This does sacrifice the two use cases listed above, but siying said it's ok.
Overview of the changes:
* A commit that adds missing status checks in MergingIterator. This fixes a bug that actually affects us, and we need it fixed. `DBIteratorTest.NonBlockingIterationBugRepro` explains the scenario.
* Changes to lots of iterator types to make all of them conform to the new convention. Some bug fixes along the way. By far the biggest changes are in DBIter, which is a big messy piece of code; I tried to make it less big and messy but mostly failed.
* A stress-test for DBIter, to gain some confidence that I didn't break it. It does a few million random operations on the iterator, while occasionally modifying the underlying data (like ForwardIterator does) and occasionally returning non-ok status from internal iterator.
To find the iterator types that needed changes I searched for "public .*Iterator" in the code. Here's an overview of all 27 iterator types:
Iterators that didn't need changes:
* status() is always ok(), or Valid() is always false: MemTableIterator, ModelIter, TestIterator, KVIter (2 classes with this name anonymous namespaces), LoggingForwardVectorIterator, VectorIterator, MockTableIterator, EmptyIterator, EmptyInternalIterator.
* Thin wrappers that always pass through Valid() and status(): ArenaWrappedDBIter, TtlIterator, InternalIteratorFromIterator.
Iterators with changes (see inline comments for details):
* DBIter - an overhaul:
- It used to silently skip corrupted keys (`FindParseableKey()`), which seems dangerous. This PR makes it just stop immediately after encountering a corrupted key, just like it would for other kinds of corruption. Let me know if there was actually some deeper meaning in this behavior and I should put it back.
- It had a few code paths silently discarding subiterator's status. The stress test caught a few.
- The backwards iteration code path was expecting the internal iterator's set of keys to be immutable. It's probably always true in practice at the moment, since ForwardIterator doesn't support backwards iteration, but this PR fixes it anyway. See added DBIteratorTest.ReverseToForwardBug for an example.
- Some parts of backwards iteration code path even did things like `assert(iter_->Valid())` after a seek, which is never a safe assumption.
- It used to not reset status on seek for some types of errors.
- Some simplifications and better comments.
- Some things got more complicated from the added error handling. I'm open to ideas for how to make it nicer.
* MergingIterator - check status after every operation on every subiterator, and in some places assert that valid subiterators have ok status.
* ForwardIterator - changed to the new convention, also slightly simplified.
* ForwardLevelIterator - fixed some bugs and simplified.
* LevelIterator - simplified.
* TwoLevelIterator - changed to the new convention. Also fixed a bug that would make SeekForPrev() sometimes silently ignore errors from first_level_iter_.
* BlockBasedTableIterator - minor changes.
* BlockIter - replaced `SetStatus()` with `Invalidate()` to make sure non-ok BlockIter is always invalid.
* PlainTableIterator - some seeks used to not reset status.
* CuckooTableIterator - tiny code cleanup.
* ManagedIterator - fixed some bugs.
* BaseDeltaIterator - changed to the new convention and fixed a bug.
* BlobDBIterator - seeks used to not reset status.
* KeyConvertingIterator - some small change.
Closes https://github.com/facebook/rocksdb/pull/3810
Differential Revision: D7888019
Pulled By: al13n321
fbshipit-source-id: 4aaf6d3421c545d16722a815b2fa2e7912bc851d
7 years ago
|
|
|
// REQUIRES: Valid()
|
|
|
|
virtual Slice value() const = 0;
|
|
|
|
|
|
|
|
// Return the wide columns for the current entry. If the entry is a
|
|
|
|
// wide-column entity, return it as-is; if it is a plain key-value, return it
|
|
|
|
// as an entity with a single anonymous column (see kDefaultWideColumnName)
|
|
|
|
// which contains the value. The underlying storage for the returned
|
|
|
|
// structure is valid only until the next modification of the iterator (i.e.
|
|
|
|
// the next SeekToFirst/SeekToLast/Seek/SeekForPrev/Next/Prev operation).
|
|
|
|
// REQUIRES: Valid()
|
|
|
|
virtual const WideColumns& columns() const {
|
|
|
|
assert(false);
|
|
|
|
return kNoWideColumns;
|
|
|
|
}
|
|
|
|
|
|
|
|
// If an error has occurred, return it. Else return an ok status.
|
|
|
|
// If non-blocking IO is requested and this operation cannot be
|
|
|
|
// satisfied without doing some IO, then this returns Status::Incomplete().
|
|
|
|
virtual Status status() const = 0;
|
|
|
|
|
|
|
|
// If supported, renew the iterator to represent the latest state. The
|
|
|
|
// iterator will be invalidated after the call. Not supported if
|
|
|
|
// ReadOptions.snapshot is given when creating the iterator.
|
|
|
|
virtual Status Refresh() {
|
|
|
|
return Status::NotSupported("Refresh() is not supported");
|
|
|
|
}
|
|
|
|
|
|
|
|
// Property "rocksdb.iterator.is-key-pinned":
|
|
|
|
// If returning "1", this means that the Slice returned by key() is valid
|
|
|
|
// as long as the iterator is not deleted.
|
|
|
|
// It is guaranteed to always return "1" if
|
|
|
|
// - Iterator created with ReadOptions::pin_data = true
|
|
|
|
// - DB tables were created with
|
|
|
|
// BlockBasedTableOptions::use_delta_encoding = false.
|
|
|
|
// Property "rocksdb.iterator.super-version-number":
|
|
|
|
// LSM version used by the iterator. The same format as DB Property
|
|
|
|
// kCurrentSuperVersionNumber. See its comment for more information.
|
|
|
|
// Property "rocksdb.iterator.internal-key":
|
|
|
|
// Get the user-key portion of the internal key at which the iteration
|
|
|
|
// stopped.
|
|
|
|
virtual Status GetProperty(std::string prop_name, std::string* prop);
|
|
|
|
|
|
|
|
virtual Slice timestamp() const {
|
|
|
|
assert(false);
|
|
|
|
return Slice();
|
|
|
|
}
|
|
|
|
};
|
|
|
|
|
|
|
|
// Return an empty iterator (yields nothing).
|
|
|
|
extern Iterator* NewEmptyIterator();
|
|
|
|
|
|
|
|
// Return an empty iterator with the specified status.
|
|
|
|
extern Iterator* NewErrorIterator(const Status& status);
|
|
|
|
|
|
|
|
} // namespace ROCKSDB_NAMESPACE
|