fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Peter Dillinger 2a383f21f4 Add Bloom/Ribbon hybrid API support (#8679) ... Summary: This is essentially resurrection and fixing of the part of https://github.com/facebook/rocksdb/issues/8198 that was reverted in https://github.com/facebook/rocksdb/issues/8212, using data added in https://github.com/facebook/rocksdb/issues/8246. Basically, when configuring Ribbon filter, you can specify an LSM level before which Bloom will be used instead of Ribbon. But Bloom is only considered for Leveled and Universal compaction styles and file going into a known LSM level. This way, SST file writer, FIFO compaction, etc. use Ribbon filter as you would expect with NewRibbonFilterPolicy. So that this can be controlled with a single int value and so that flushes can be distinguished from intra-L0, we consider flush to go to level -1 for the purposes of this option. (Explained in API comment.) I also expect the most common and recommended Ribbon configuration to use Bloom during flush, to minimize slowing down writes and because according to my estimates, Ribbon only pays off if the structure lives in memory for more than an hour. Thus, I have changed the default for NewRibbonFilterPolicy to be this mild hybrid configuration. I don't really want to add something like NewHybridFilterPolicy because at least the mild hybrid configuration (Bloom for flush, Ribbon otherwise) should be considered a natural choice. C APIs also updated, but because they don't support overloading, rocksdb_filterpolicy_create_ribbon is kept pure ribbon for clarity and rocksdb_filterpolicy_create_ribbon_hybrid must be called for a hybrid configuration. While touching C API, I changed bits per key options from int to double. BuiltinFilterPolicy is needed so that LevelThresholdFilterPolicy doesn't inherit unused fields from BloomFilterPolicy. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8679 Test Plan: new + updated tests, including crash test Reviewed By: jay-zhuang Differential Revision: D30445797 Pulled By: pdillinger fbshipit-source-id: 6f5aeddfd6d79f7e55493b563c2d1d2d568892e1		3 years ago
..
blob	Add statistics support to integrated BlobDB (#8667)	3 years ago
compaction	Fix bug caused by releasing snapshot(s) during compaction (#8608)	3 years ago
db_impl	Fix blob callback in compaction and atomic flush (#8681)	3 years ago
arena_wrapped_db_iter.cc	Rename ImmutableOptions variables (#8409)	3 years ago
arena_wrapped_db_iter.h	Rename ImmutableOptions variables (#8409)	3 years ago
builder.cc	Memtable "MemPurge" prototype (#8454)	3 years ago
builder.h	Added memtable garbage statistics (#8411)	3 years ago
c.cc	Add Bloom/Ribbon hybrid API support (#8679)	3 years ago
c_test.c	Add Bloom/Ribbon hybrid API support (#8679)	3 years ago
column_family.cc	Fix a race in ColumnFamilyData::UnrefAndTryDelete (#8605)	3 years ago
column_family.h	Fix a race in ColumnFamilyData::UnrefAndTryDelete (#8605)	3 years ago
column_family_test.cc	Add CreateFrom methods to Env/FileSystem (#8174)	3 years ago
compact_files_test.cc	Compaction should not move data to up level (#8116)	4 years ago
comparator_db_test.cc	Remove unused includes (#7604)	4 years ago
convenience.cc	Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262)	4 years ago
corruption_test.cc	Add CreateFrom methods to Env/FileSystem (#8174)	3 years ago
cuckoo_table_db_test.cc	Revert "Turn on memtable bloom filter by default. (#6584)" (#7939)	4 years ago
db_basic_test.cc	Stable cache keys using DB session ids in SSTs (#8659)	3 years ago
db_block_cache_test.cc	Stable cache keys on ingested SST files (#8669)	3 years ago
db_bloom_filter_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_compaction_filter_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_compaction_test.cc	Move old files to warm tier in FIFO compactions (#8310)	3 years ago
db_dynamic_level_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_encryption_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_filesnapshot.cc	DB::GetSortedWalFiles() to ensure file deletion is disabled (#8591)	3 years ago
db_flush_test.cc	Add condition on NotifyOnFlushComplete that FlushJob was not mempurge. Add event listeners to mempurge tests. (#8672)	3 years ago
db_info_dumper.cc	Allow WAL dir to change with db dir (#8582)	3 years ago
db_info_dumper.h	Add a DB Session ID (#6959)	4 years ago
db_inplace_update_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_io_failure_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_iter.cc	Rename ImmutableOptions variables (#8409)	3 years ago
db_iter.h	Rename ImmutableOptions variables (#8409)	3 years ago
db_iter_stress_test.cc	Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262)	4 years ago
db_iter_test.cc	Rename ImmutableOptions variables (#8409)	3 years ago
db_iterator_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_kv_checksum_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_log_iter_test.cc	Attempt to deflake DBTestXactLogIterator.TransactionLogIteratorCorruptedLog (#8627)	3 years ago
db_logical_block_size_cache_test.cc	Add further tests to ASSERT_STATUS_CHECKED (2) (#7698)	4 years ago
db_memtable_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_merge_operand_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_merge_operator_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_options_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_properties_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_range_del_test.cc	Fix missing Handle release in TableCache::GetRangeTombstoneIterator (#8589)	3 years ago
db_secondary_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_sst_test.cc	Fix blob callback in compaction and atomic flush (#8681)	3 years ago
db_statistics_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_table_properties_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_tailing_iter_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_test2.cc	Add `IteratorTraceExecutionResult` for iterator related trace records. (#8687)	3 years ago
db_test_util.cc	Make EncryptionProvider and BlockCipher into Customizable objects (#8354)	3 years ago
db_test_util.h	Add db_test2 to to ASSERT_STATUS_CHECKED (#8640)	3 years ago
db_universal_compaction_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_wal_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_with_timestamp_basic_test.cc	Move slow valgrind tests behind -DROCKSDB_FULL_VALGRIND_RUN (#8475)	3 years ago
db_with_timestamp_compaction_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_write_buffer_manager_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_write_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
dbformat.cc	Make CompactRange and GetApproximateSizes work with timestamp (#7684)	4 years ago
dbformat.h	Fix some typos in comments (#8066)	4 years ago
dbformat_test.cc	Remove unused includes (#7604)	4 years ago
deletefile_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
error_handler.cc	DB::GetSortedWalFiles() to ensure file deletion is disabled (#8591)	3 years ago
error_handler.h	Fix some typos in comments (#8066)	4 years ago
error_handler_fs_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
event_helpers.cc	Make EventListener into a Customizable Class (#8473)	3 years ago
event_helpers.h	Pass SST file checksum information through OnTableFileCreated (#7108)	4 years ago
experimental.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
external_sst_file_basic_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
external_sst_file_ingestion_job.cc	Sync ingested files only if reopen is supported by the FS (#8296)	4 years ago
external_sst_file_ingestion_job.h	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033)	4 years ago
external_sst_file_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
fault_injection_test.cc	Add more tests for assert status checked (#7524)	4 years ago
file_indexer.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
file_indexer.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
file_indexer_test.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
filename_test.cc	Remove unused includes (#7604)	4 years ago
flush_job.cc	Add condition on NotifyOnFlushComplete that FlushJob was not mempurge. Add event listeners to mempurge tests. (#8672)	3 years ago
flush_job.h	Add condition on NotifyOnFlushComplete that FlushJob was not mempurge. Add event listeners to mempurge tests. (#8672)	3 years ago
flush_job_test.cc	Fix NotifyOnFlushCompleted() for atomic flush (#8585)	3 years ago
flush_scheduler.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
flush_scheduler.h	Include C++ standard library headers instead of C compatibility headers (#8068)	4 years ago
forward_iterator.cc	Fix some typos in comments (#8066)	4 years ago
forward_iterator.h	Properly report IO errors when IndexType::kBinarySearchWithFirstKey is used (#6621)	5 years ago
forward_iterator_bench.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
import_column_family_job.cc	Add a SystemClock class to capture the time functions of an Env (#7858)	4 years ago
import_column_family_job.h	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033)	4 years ago
import_column_family_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
internal_stats.cc	Add property `LiveSstFilesSizeAtTemperature` for tiered storage (#8644)	3 years ago
internal_stats.h	Add property `LiveSstFilesSizeAtTemperature` for tiered storage (#8644)	3 years ago
job_context.h	Rename ImmutableOptions variables (#8409)	3 years ago
kv_checksum.h	Integrity protection for live updates to WriteBatch (#7748)	4 years ago
listener_test.cc	Fix NotifyOnFlushCompleted() for atomic flush (#8585)	3 years ago
log_format.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
log_reader.cc	Fix kPointInTimeRecovery handling of truncated WAL (#7701)	4 years ago
log_reader.h	Real fix for race in backup custom checksum checking (#7309)	4 years ago
log_test.cc	Make StringEnv, StringSink, StringSource use FS classes (#7786)	4 years ago
log_writer.cc	Using existing crc32c checksum in checksum handoff for Manifest and WAL (#8412)	3 years ago
log_writer.h	Include C++ standard library headers instead of C compatibility headers (#8068)	4 years ago
logs_with_prep_tracker.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
logs_with_prep_tracker.h	Include C++ standard library headers instead of C compatibility headers (#8068)	4 years ago
lookup_key.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
malloc_stats.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
malloc_stats.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
manual_compaction_test.cc	Add more tests for assert status checked (#7524)	4 years ago
memtable.cc	Retire superfluous functions introduced in earlier mempurge PRs. (#8558)	3 years ago
memtable.h	Memtable sampling for mempurge heuristic. (#8628)	3 years ago
memtable_list.cc	Improve MemPurge sampling (#8656)	3 years ago
memtable_list.h	Improve MemPurge sampling (#8656)	3 years ago
memtable_list_test.cc	Fix NotifyOnFlushCompleted() for atomic flush (#8585)	3 years ago
merge_context.h	Add Merge Operator support to WriteBatchWithIndex (#8135)	4 years ago
merge_helper.cc	Add support for Merge with base value during Compaction in IntegratedBlobDB (#8445)	3 years ago
merge_helper.h	Add support for Merge with base value during Compaction in IntegratedBlobDB (#8445)	3 years ago
merge_helper_test.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
merge_operator.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
merge_test.cc	MergeHelper::FilterMerge() calling ElapsedNanosSafe() upon exit even … (#7867)	4 years ago
obsolete_files_test.cc	Attempt to deflake ObsoleteFilesTest.DeleteObsoleteOptionsFile (#8624)	3 years ago
options_file_test.cc	No elide constructors (#7798)	4 years ago
output_validator.cc	Use NPHash64 in more places (#7632)	4 years ago
output_validator.h	Add remote compaction public API (#8300)	4 years ago
perf_context_test.cc	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033)	4 years ago
periodic_work_scheduler.cc	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033)	4 years ago
periodic_work_scheduler.h	Add a SystemClock class to capture the time functions of an Env (#7858)	4 years ago
periodic_work_scheduler_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
pinned_iterators_manager.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
plain_table_db_test.cc	Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262)	4 years ago
pre_release_callback.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
prefix_test.cc	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033)	4 years ago
range_del_aggregator.cc	In ParseInternalKey(), include corrupt key info in Status (#7515)	4 years ago
range_del_aggregator.h	Fix some typos in comments (#8066)	4 years ago
range_del_aggregator_bench.cc	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033)	4 years ago
range_del_aggregator_test.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
range_tombstone_fragmenter.cc	Added memtable garbage statistics (#8411)	3 years ago
range_tombstone_fragmenter.h	Added memtable garbage statistics (#8411)	3 years ago
range_tombstone_fragmenter_test.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
read_callback.h	Get() with timestamp should respect snapshot (#7227)	4 years ago
repair.cc	Stable cache keys on ingested SST files (#8669)	3 years ago
repair_test.cc	Some fixes and enhancements to `ldb repair` (#8544)	3 years ago
snapshot_checker.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
snapshot_impl.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
snapshot_impl.h	Fix some typos in comments (#8066)	4 years ago
table_cache.cc	Fix use-after-free on implicit temporary FileOptions (#8571)	3 years ago
table_cache.h	Fix use-after-free on implicit temporary FileOptions (#8571)	3 years ago
table_properties_collector.cc	Apply `sample_for_compression` to all block-based tables (#8105)	4 years ago
table_properties_collector.h	Partially revert the "apply subrange of table property collectors" change (#8465)	3 years ago
table_properties_collector_test.cc	Make it possible to apply only a subrange of table property collectors (#8298)	4 years ago
transaction_log_impl.cc	Add more tests for assert status checked (#7524)	4 years ago
transaction_log_impl.h	Store FileSystemPtr object that contains FileSystem ptr (#7180)	4 years ago
trim_history_scheduler.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
trim_history_scheduler.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
version_builder.cc	Handle blob files when options.best_efforts_recovery is true (#8180)	4 years ago
version_builder.h	Handle blob files when options.best_efforts_recovery is true (#8180)	4 years ago
version_builder_test.cc	Print blob file checksums as hex (#8437)	3 years ago
version_edit.cc	Write file temperature information to manifest (#8284)	4 years ago
version_edit.h	Write file temperature information to manifest (#8284)	4 years ago
version_edit_handler.cc	Retire superfluous functions introduced in earlier mempurge PRs. (#8558)	3 years ago
version_edit_handler.h	Fixed manifest_dump issues when printing keys and values containing null characters (#8378)	3 years ago
version_edit_test.cc	Make it able to ignore WAL related VersionEdits in older versions (#7873)	4 years ago
version_set.cc	Move old files to warm tier in FIFO compactions (#8310)	3 years ago
version_set.h	Retire superfluous functions introduced in earlier mempurge PRs. (#8558)	3 years ago
version_set_test.cc	Print blob file checksums as hex (#8437)	3 years ago
wal_edit.cc	Always track WAL obsoletion (#7759)	4 years ago
wal_edit.h	Always track WAL obsoletion (#7759)	4 years ago
wal_edit_test.cc	Always track WAL obsoletion (#7759)	4 years ago
wal_manager.cc	Allow WAL dir to change with db dir (#8582)	3 years ago
wal_manager.h	Allow WAL dir to change with db dir (#8582)	3 years ago
wal_manager_test.cc	Use DbSessionId as cache key prefix when secondary cache is enabled (#8360)	3 years ago
write_batch.cc	Several simple local code clean-ups (#8565)	3 years ago
write_batch_base.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
write_batch_internal.h	Several simple local code clean-ups (#8565)	3 years ago
write_batch_test.cc	Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262)	4 years ago
write_callback.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
write_callback_test.cc	Move slow valgrind tests behind -DROCKSDB_FULL_VALGRIND_RUN (#8475)	3 years ago
write_controller.cc	Revamp WriteController (#8064)	4 years ago
write_controller.h	Revamp WriteController (#8064)	4 years ago
write_controller_test.cc	Revamp WriteController (#8064)	4 years ago
write_thread.cc	Stall writes in WriteBufferManager when memory_usage exceeds buffer_size (#7898)	4 years ago
write_thread.h	typo: fix typo in db/write_thread's state (#8423)	3 years ago