fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Peter Dillinger 003e72b201 Use size_t for filter APIs, protect against overflow (#7726) ... Summary: Deprecate CalculateNumEntry and replace with ApproximateNumEntries (better name) using size_t instead of int and uint32_t, to minimize confusing casts and bad overflow behavior (possible though probably not realistic). Bloom sizes are now explicitly capped at max size supported by implementations: just under 4GiB for fv=5 Bloom, and just under 512MiB for fv<5 Legacy Bloom. This hardening could help to set up for fuzzing. Also, since RocksDB only uses this information as an approximation for trying to hit certain sizes for partitioned filters, it's more important that the function be reasonably fast than for it to be completely accurate. It's hard enough to be 100% accurate for Ribbon (currently reversing CalculateSpace) that adding optimize_filters_for_memory into the mix is just not worth trying to be 100% accurate for num entries for bytes. Also: - Cleaned up filter_policy.h to remove MSVC warning handling and potentially unsafe use of exception for "not implemented" - Correct the number of entries limit beyond which current Ribbon implementation falls back on Bloom instead. - Consistently use "num_entries" rather than "num_entry" - Remove LegacyBloomBitsBuilder::CalculateNumEntry as it's essentially obsolete from general implementation BuiltinFilterBitsBuilder::CalculateNumEntries. - Fix filter_bench to skip some tests that don't make sense when only one or a small number of filters has been generated. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7726 Test Plan: expanded existing unit tests for CalculateSpace / ApproximateNumEntries. Also manually used filter_bench to verify Legacy and fv=5 Bloom size caps work (much too expensive for unit test). Note that the actual bits per key is below requested due to space cap. $ ./filter_bench -impl=0 -bits_per_key=20 -average_keys_per_filter=256000000 -vary_key_count_ratio=0 -m_keys_total_max=256 -allow_bad_fp_rate ... Total size (MB): 511.992 Bits/key stored: 16.777 ... $ ./filter_bench -impl=2 -bits_per_key=20 -average_keys_per_filter=2000000000 -vary_key_count_ratio=0 -m_keys_total_max=2000 ... Total size (MB): 4096 Bits/key stored: 17.1799 ... $ Reviewed By: jay-zhuang Differential Revision: D25239800 Pulled By: pdillinger fbshipit-source-id: f94e6d065efd31e05ec630ae1a82e6400d8390c4		4 years ago
..
adaptive	Bring the Configurable options together (#5753)	4 years ago
block_based	Use size_t for filter APIs, protect against overflow (#7726)	4 years ago
cuckoo	Create a Customizable class to load classes and configurations (#6590)	4 years ago
plain	Create a Customizable class to load classes and configurations (#6590)	4 years ago
block_fetcher.cc	Skip unnecessary allocation for mmap reads under 5000 bytes (#7043)	4 years ago
block_fetcher.h	Reduce memory copies when fetching and uncompressing blocks from SST files (#6689)	5 years ago
block_fetcher_test.cc	Add Status check enforcement for unit tests (#7464)	4 years ago
cleanable_test.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
format.cc	Fix initialization order of DBOptions and kHostnameForDbHostId (#7702)	4 years ago
format.h	Add a host location property to TableProperties (#7479)	4 years ago
get_context.cc	Introduce BlobFileCache and add support for blob files to Get() (#7540)	4 years ago
get_context.h	Introduce BlobFileCache and add support for blob files to Get() (#7540)	4 years ago
internal_iterator.h	Clean up InternalIterator upper bound logic a little bit (#7200)	4 years ago
iter_heap.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
iterator.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
iterator_wrapper.h	Redesign block cache pinning API (#7520)	4 years ago
merger_test.cc	More Makefile Cleanup (#7097)	4 years ago
merging_iterator.cc	Add some simulator cache and block tracer tests to ASSERT_STATUS_CHECKED (#7305)	4 years ago
merging_iterator.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
meta_blocks.cc	Add a host location property to TableProperties (#7479)	4 years ago
meta_blocks.h	Extend Get/MultiGet deadline support to table open (#6982)	4 years ago
mock_table.cc	Add full_history_ts_low_ to CompactionJob (#7657)	4 years ago
mock_table.h	Add full_history_ts_low_ to CompactionJob (#7657)	4 years ago
multiget_context.h	Fix MultiGet unable to query timestamp data issue (#7589)	4 years ago
persistent_cache_helper.cc	Add more tests to ASSERT_STATUS_CHECKED (#7367)	4 years ago
persistent_cache_helper.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
persistent_cache_options.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
scoped_arena_iterator.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
sst_file_dumper.cc	In ParseInternalKey(), include corrupt key info in Status (#7515)	4 years ago
sst_file_dumper.h	Add sst_file_dumper status check (#7315)	4 years ago
sst_file_reader.cc	Add blob support to DBIter (#7731)	4 years ago
sst_file_reader_test.cc	Add a host location property to TableProperties (#7479)	4 years ago
sst_file_writer.cc	enable Status check assertions for sst_file_reader_test (#7448)	4 years ago
sst_file_writer_collectors.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
table_builder.h	Store DB identity and DB session ID in SST files (#6983)	4 years ago
table_factory.cc	Create a Customizable class to load classes and configurations (#6590)	4 years ago
table_properties.cc	Add a host location property to TableProperties (#7479)	4 years ago
table_properties_internal.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
table_reader.h	dedup ReadOptions in iterator hierarchy (#7210)	4 years ago
table_reader_bench.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
table_reader_caller.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
table_test.cc	Return `Status` from `MemTable` mutation functions (#7656)	4 years ago
two_level_iterator.cc	Redesign block cache pinning API (#7520)	4 years ago
two_level_iterator.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago