fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Peter Dillinger 479eb1aad6 Hide deprecated, inefficient block-based filter from public API (#9535) ... Summary: This change removes the ability to configure the deprecated, inefficient block-based filter in the public API. Options that would have enabled it now use "full" (and optionally partitioned) filters. Existing block-based filters can still be read and used, and a "back door" way to build them still exists, for testing and in case of trouble. About the only way this removal would cause an issue for users is if temporary memory for filter construction greatly increases. In HISTORY.md we suggest a few possible mitigations: partitioned filters, smaller SST files, or setting reserve_table_builder_memory=true. Or users who have customized a FilterPolicy using the CreateFilter/KeyMayMatch mechanism removed in https://github.com/facebook/rocksdb/issues/9501 will have to upgrade their code. (It's long past time for people to move to the new builder/reader customization interface.) This change also introduces some internal-use-only configuration strings for testing specific filter implementations while bypassing some compatibility / intelligence logic. This is intended to hint at a path toward making FilterPolicy Customizable, but it also gives us a "back door" way to configure block-based filter. Aside: updated db_bench so that -readonly implies -use_existing_db Pull Request resolved: https://github.com/facebook/rocksdb/pull/9535 Test Plan: Unit tests updated. Specifically, * BlockBasedTableTest.BlockReadCountTest is tweaked to validate the back door configuration interface and ignoring of `use_block_based_builder`. * BlockBasedTableTest.TracingGetTest is migrated from testing block-based filter access pattern to full filter access patter, by re-ordering some things. * Options test (pretty self-explanatory) Performance test - create with `./db_bench -db=/dev/shm/rocksdb1 -bloom_bits=10 -cache_index_and_filter_blocks=1 -benchmarks=fillrandom -num=10000000 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=0` with and without `-use_block_based_filter`, which creates a DB with 21 SST files in L0. Read with `./db_bench -db=/dev/shm/rocksdb1 -readonly -bloom_bits=10 -cache_index_and_filter_blocks=1 -benchmarks=readrandom -num=10000000 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=0 -duration=30` Without -use_block_based_filter: readrandom 464 ops/sec, 689280 KB DB With -use_block_based_filter: readrandom 169 ops/sec, 690996 KB DB No consistent difference with fillrandom Reviewed By: jay-zhuang Differential Revision: D34153871 Pulled By: pdillinger fbshipit-source-id: 31f4a933c542f8f09aca47fa64aec67832a69738		3 years ago
..
blob	Log blob file space amp and expose it via the rocksdb.blob-stats DB property (#9538)	3 years ago
compaction	Use a sorted vector instead of a map to store blob file metadata (#9526)	3 years ago
db_impl	Avoid unnecessary copy of sample_slice map (#9551)	3 years ago
arena_wrapped_db_iter.cc	Rename ImmutableOptions variables (#8409)	3 years ago
arena_wrapped_db_iter.h	Cleanup includes in dbformat.h (#8930)	3 years ago
builder.cc	Fast path for detecting unchanged prefix_extractor (#9407)	3 years ago
builder.h	Expose blob file information through the EventListener interface (#8675)	3 years ago
c.cc	Fix failure in c_test (#9547)	3 years ago
c_test.c	Hide deprecated, inefficient block-based filter from public API (#9535)	3 years ago
column_family.cc	Fix bug affecting GetSortedWalFiles, Backups, Checkpoint (#9208)	3 years ago
column_family.h	Add DB properties for BlobDB (#8734)	3 years ago
column_family_test.cc	More refactoring ahead of footer & meta changes (#9240)	3 years ago
compact_files_test.cc	Compaction should not move data to up level (#8116)	4 years ago
comparator_db_test.cc	More refactoring ahead of footer & meta changes (#9240)	3 years ago
convenience.cc	Fast path for detecting unchanged prefix_extractor (#9407)	3 years ago
corruption_test.cc	Make the Env class Customizable (#9293)	3 years ago
cuckoo_table_db_test.cc	Experimental support for SST unique IDs (#8990)	3 years ago
db_basic_test.cc	Disallow a combination of options (#9348)	3 years ago
db_block_cache_test.cc	Enhance new cache key testing & comments (#9329)	3 years ago
db_bloom_filter_test.cc	FilterPolicy API changes for 7.0 (#9501)	3 years ago
db_compaction_filter_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_compaction_test.cc	Use a sorted vector instead of a map to store blob file metadata (#9526)	3 years ago
db_dynamic_level_test.cc	Remove deprecated API AdvancedColumnFamilyOptions::soft_rate_limit/hard_rate_limit (#9452)	3 years ago
db_encryption_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_filesnapshot.cc	Use a sorted vector instead of a map to store blob file metadata (#9526)	3 years ago
db_flush_test.cc	Use a sorted vector instead of a map to store blob file metadata (#9526)	3 years ago
db_info_dumper.cc	Allow WAL dir to change with db dir (#8582)	3 years ago
db_info_dumper.h	Add a DB Session ID (#6959)	4 years ago
db_inplace_update_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_io_failure_test.cc	Enable a few unit tests to use custom Env objects (#9087)	3 years ago
db_iter.cc	Remove iter_start_seqnum and preserve_deletes (#9430)	3 years ago
db_iter.h	Cleanup includes in dbformat.h (#8930)	3 years ago
db_iter_stress_test.cc	Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262)	4 years ago
db_iter_test.cc	Remove iter_start_seqnum and preserve_deletes (#9430)	3 years ago
db_iterator_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_kv_checksum_test.cc	Revise APIs related to user-defined timestamp (#8946)	3 years ago
db_log_iter_test.cc	Attempt to deflake DBTestXactLogIterator.TransactionLogIteratorCorruptedLog (#8627)	3 years ago
db_logical_block_size_cache_test.cc	Add further tests to ASSERT_STATUS_CHECKED (2) (#7698)	4 years ago
db_memtable_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_merge_operand_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_merge_operator_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_options_test.cc	Remove deprecated option new_table_reader_for_compaction_inputs (#9443)	3 years ago
db_properties_test.cc	Remove iter_start_seqnum and preserve_deletes (#9430)	3 years ago
db_range_del_test.cc	Incremental Space Amp Compactions in Universal Style (#8655)	3 years ago
db_secondary_test.cc	Make the Env class Customizable (#9293)	3 years ago
db_sst_test.cc	Make the Env class Customizable (#9293)	3 years ago
db_statistics_test.cc	Bytes read stat for `VerifyChecksum()` and `VerifyFileChecksums()` APIs (#8741)	3 years ago
db_table_properties_test.cc	Fix backward compatibility with 2.5 through 2.7 (#9189)	3 years ago
db_tailing_iter_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
db_test.cc	Fix spelling in public API (#9490)	3 years ago
db_test2.cc	Remove deprecated option new_table_reader_for_compaction_inputs (#9443)	3 years ago
db_test_util.cc	Use a sorted vector instead of a map to store blob file metadata (#9526)	3 years ago
db_test_util.h	Remove iter_start_seqnum and preserve_deletes (#9430)	3 years ago
db_universal_compaction_test.cc	Adhere to per-DB concurrency limit when bottom-pri compactions exist (#9179)	3 years ago
db_wal_test.cc	Use a sorted vector instead of a map to store blob file metadata (#9526)	3 years ago
db_with_timestamp_basic_test.cc	Use the comparator from the sst file table properties in sst_dump_tool (#9491)	3 years ago
db_with_timestamp_compaction_test.cc	Use the comparator from the sst file table properties in sst_dump_tool (#9491)	3 years ago
db_write_buffer_manager_test.cc	Enable a few unit tests to use custom Env objects (#9087)	3 years ago
db_write_test.cc	Enable a few unit tests to use custom Env objects (#9087)	3 years ago
dbformat.cc	Track per-SST user-defined timestamp information in MANIFEST (#9092)	3 years ago
dbformat.h	Remove timestamp from key in expected state (#9525)	3 years ago
dbformat_test.cc	Enable a few unit tests to use custom Env objects (#9087)	3 years ago
deletefile_test.cc	Enable a few unit tests to use custom Env objects (#9087)	3 years ago
error_handler.cc	Work around some new clang-analyze failures (#9515)	3 years ago
error_handler.h	cleanup error_handler related code (#9098)	3 years ago
error_handler_fs_test.cc	Fix race condition in `error_handler_fs_test` (#9325)	3 years ago
event_helpers.cc	Add a listener callback for end of auto error recovery (#9244)	3 years ago
event_helpers.h	Add a listener callback for end of auto error recovery (#9244)	3 years ago
experimental.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
external_sst_file_basic_test.cc	Enable a few unit tests to use custom Env objects (#9087)	3 years ago
external_sst_file_ingestion_job.cc	Fast path for detecting unchanged prefix_extractor (#9407)	3 years ago
external_sst_file_ingestion_job.h	New stable, fixed-length cache keys (#9126)	3 years ago
external_sst_file_test.cc	Make the Env class Customizable (#9293)	3 years ago
fault_injection_test.cc	Fix a bug causing duplicate trailing entries in WritableFile (buffered IO) (#9236)	3 years ago
file_indexer.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
file_indexer.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
file_indexer_test.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
filename_test.cc	Remove unused includes (#7604)	4 years ago
flush_job.cc	Use a sorted vector instead of a map to store blob file metadata (#9526)	3 years ago
flush_job.h	Cleanup includes in dbformat.h (#8930)	3 years ago
flush_job_test.cc	Use the comparator from the sst file table properties in sst_dump_tool (#9491)	3 years ago
flush_scheduler.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
flush_scheduler.h	Include C++ standard library headers instead of C compatibility headers (#8068)	4 years ago
forward_iterator.cc	Fast path for detecting unchanged prefix_extractor (#9407)	3 years ago
forward_iterator.h	Fast path for detecting unchanged prefix_extractor (#9407)	3 years ago
forward_iterator_bench.cc	Remove using namespace (#9369)	3 years ago
import_column_family_job.cc	Fast path for detecting unchanged prefix_extractor (#9407)	3 years ago
import_column_family_job.h	New stable, fixed-length cache keys (#9126)	3 years ago
import_column_family_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
internal_stats.cc	Log blob file space amp and expose it via the rocksdb.blob-stats DB property (#9538)	3 years ago
internal_stats.h	Support `GetMapProperty()` with "rocksdb.dbstats" (#9057)	3 years ago
job_context.h	Fix and detect headers with missing dependencies (#8893)	3 years ago
kv_checksum.h	fix compile errors in db/kv_checksum.h (#9173)	3 years ago
listener_test.cc	Fix TSAN data race in EventListenerTest.MultiCF (#9528)	3 years ago
log_format.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
log_reader.cc	Fix kPointInTimeRecovery handling of truncated WAL (#7701)	4 years ago
log_reader.h	Real fix for race in backup custom checksum checking (#7309)	4 years ago
log_test.cc	Make StringEnv, StringSink, StringSource use FS classes (#7786)	4 years ago
log_writer.cc	Using existing crc32c checksum in checksum handoff for Manifest and WAL (#8412)	3 years ago
log_writer.h	Include C++ standard library headers instead of C compatibility headers (#8068)	4 years ago
logs_with_prep_tracker.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
logs_with_prep_tracker.h	Include C++ standard library headers instead of C compatibility headers (#8068)	4 years ago
lookup_key.h	Cleanup includes in dbformat.h (#8930)	3 years ago
malloc_stats.cc	Replace most typedef with using= (#8751)	3 years ago
malloc_stats.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
manual_compaction_test.cc	Remove using namespace (#9369)	3 years ago
memtable.cc	Fix major bug with MultiGet, DeleteRange, and memtable Bloom (#9453)	3 years ago
memtable.h	Fix major bug with MultiGet, DeleteRange, and memtable Bloom (#9453)	3 years ago
memtable_list.cc	Make MemoryAllocator into a Customizable class (#8980)	3 years ago
memtable_list.h	Make MemoryAllocator into a Customizable class (#8980)	3 years ago
memtable_list_test.cc	MemTableList::TrimHistory now use allocated bytes (#9020)	3 years ago
merge_context.h	Add Merge Operator support to WriteBatchWithIndex (#8135)	4 years ago
merge_helper.cc	Support readahead during compaction for blob files (#9187)	3 years ago
merge_helper.h	Support readahead during compaction for blob files (#9187)	3 years ago
merge_helper_test.cc	Support readahead during compaction for blob files (#9187)	3 years ago
merge_operator.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
merge_test.cc	Make the Env class Customizable (#9293)	3 years ago
obsolete_files_test.cc	Add commit marker with timestamp (#9266)	3 years ago
options_file_test.cc	No elide constructors (#7798)	4 years ago
output_validator.cc	Cleanup includes in dbformat.h (#8930)	3 years ago
output_validator.h	Cleanup includes in dbformat.h (#8930)	3 years ago
perf_context_test.cc	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033)	4 years ago
periodic_work_scheduler.cc	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033)	4 years ago
periodic_work_scheduler.h	Add a SystemClock class to capture the time functions of an Env (#7858)	4 years ago
periodic_work_scheduler_test.cc	Fix a minor issue with initializing the test path (#8555)	3 years ago
pinned_iterators_manager.h	Replace most typedef with using= (#8751)	3 years ago
plain_table_db_test.cc	Fast path for detecting unchanged prefix_extractor (#9407)	3 years ago
pre_release_callback.h	Fix and detect headers with missing dependencies (#8893)	3 years ago
prefix_test.cc	Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033)	4 years ago
range_del_aggregator.cc	In ParseInternalKey(), include corrupt key info in Status (#7515)	4 years ago
range_del_aggregator.h	Fix some typos in comments (#8066)	4 years ago
range_del_aggregator_bench.cc	Cleanup multiple implementations of VectorIterator (#8901)	3 years ago
range_del_aggregator_test.cc	Cleanup multiple implementations of VectorIterator (#8901)	3 years ago
range_tombstone_fragmenter.cc	Added memtable garbage statistics (#8411)	3 years ago
range_tombstone_fragmenter.h	Added memtable garbage statistics (#8411)	3 years ago
range_tombstone_fragmenter_test.cc	Cleanup multiple implementations of VectorIterator (#8901)	3 years ago
read_callback.h	Fix and detect headers with missing dependencies (#8893)	3 years ago
repair.cc	Fast path for detecting unchanged prefix_extractor (#9407)	3 years ago
repair_test.cc	Some fixes and enhancements to `ldb repair` (#8544)	3 years ago
snapshot_checker.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
snapshot_impl.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
snapshot_impl.h	Fix and detect headers with missing dependencies (#8893)	3 years ago
table_cache.cc	Fast path for detecting unchanged prefix_extractor (#9407)	3 years ago
table_cache.h	Fast path for detecting unchanged prefix_extractor (#9407)	3 years ago
table_properties_collector.cc	Apply `sample_for_compression` to all block-based tables (#8105)	4 years ago
table_properties_collector.h	Track each SST's timestamp information as user properties (#9093)	3 years ago
table_properties_collector_test.cc	Improve / clean up meta block code & integrity (#9163)	3 years ago
transaction_log_impl.cc	Add commit marker with timestamp (#9266)	3 years ago
transaction_log_impl.h	Cleanup includes in dbformat.h (#8930)	3 years ago
trim_history_scheduler.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
trim_history_scheduler.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
version_builder.cc	Use a sorted vector instead of a map to store blob file metadata (#9526)	3 years ago
version_builder.h	Fast path for detecting unchanged prefix_extractor (#9407)	3 years ago
version_builder_test.cc	Use a sorted vector instead of a map to store blob file metadata (#9526)	3 years ago
version_edit.cc	Print file checksum in hex (#9196)	3 years ago
version_edit.h	Clean up VersionStorageInfo a bit (#9494)	3 years ago
version_edit_handler.cc	Clean up VersionStorageInfo a bit (#9494)	3 years ago
version_edit_handler.h	Fixed manifest_dump issues when printing keys and values containing null characters (#8378)	3 years ago
version_edit_test.cc	File temperature information should be preserved when restart the DB (#9242)	3 years ago
version_set.cc	Rework VersionStorageInfo::ComputeFilesMarkedForForcedBlobGC a bit (#9548)	3 years ago
version_set.h	Log blob file space amp and expose it via the rocksdb.blob-stats DB property (#9538)	3 years ago
version_set_test.cc	Rework VersionStorageInfo::ComputeFilesMarkedForForcedBlobGC a bit (#9548)	3 years ago
wal_edit.cc	Always track WAL obsoletion (#7759)	4 years ago
wal_edit.h	Always track WAL obsoletion (#7759)	4 years ago
wal_edit_test.cc	Always track WAL obsoletion (#7759)	4 years ago
wal_manager.cc	Allow WAL dir to change with db dir (#8582)	3 years ago
wal_manager.h	Allow WAL dir to change with db dir (#8582)	3 years ago
wal_manager_test.cc	Make SystemClock into a Customizable Class (#8636)	3 years ago
write_batch.cc	Revise APIs related to user-defined timestamp (#8946)	3 years ago
write_batch_base.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
write_batch_internal.h	Revise APIs related to user-defined timestamp (#8946)	3 years ago
write_batch_test.cc	Use the comparator from the sst file table properties in sst_dump_tool (#9491)	3 years ago
write_callback.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
write_callback_test.cc	Move slow valgrind tests behind -DROCKSDB_FULL_VALGRIND_RUN (#8475)	3 years ago
write_controller.cc	Revamp WriteController (#8064)	4 years ago
write_controller.h	Revamp WriteController (#8064)	4 years ago
write_controller_test.cc	Revamp WriteController (#8064)	4 years ago
write_thread.cc	Stall writes in WriteBufferManager when memory_usage exceeds buffer_size (#7898)	4 years ago
write_thread.h	typo: fix typo in db/write_thread's state (#8423)	3 years ago