fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Levi Tamasi 29ccf2075c Store the filter bits reader alongside the filter block contents (#5936) ... Summary: Amongst other things, PR https://github.com/facebook/rocksdb/issues/5504 refactored the filter block readers so that only the filter block contents are stored in the block cache (as opposed to the earlier design where the cache stored the filter block reader itself, leading to potentially dangling pointers and concurrency bugs). However, this change introduced a performance hit since with the new code, the metadata fields are re-parsed upon every access. This patch reunites the block contents with the filter bits reader to eliminate this overhead; since this is still a self-contained pure data object, it is safe to store it in the cache. (Note: this is similar to how the zstd digest is handled.) Pull Request resolved: https://github.com/facebook/rocksdb/pull/5936 Test Plan: make asan_check filter_bench results for the old code: ``` $ ./filter_bench -quick WARNING: Assertions are enabled; benchmarks unnecessarily slow Building... Build avg ns/key: 26.7153 Number of filters: 16669 Total memory (MB): 200.009 Bits/key actual: 10.0647 ---------------------------- Inside queries... Dry run (46b) ns/op: 33.4258 Single filter ns/op: 42.5974 Random filter ns/op: 217.861 ---------------------------- Outside queries... Dry run (25d) ns/op: 32.4217 Single filter ns/op: 50.9855 Random filter ns/op: 219.167 Average FP rate %: 1.13993 ---------------------------- Done. (For more info, run with -legend or -help.) $ ./filter_bench -quick -use_full_block_reader WARNING: Assertions are enabled; benchmarks unnecessarily slow Building... Build avg ns/key: 26.5172 Number of filters: 16669 Total memory (MB): 200.009 Bits/key actual: 10.0647 ---------------------------- Inside queries... Dry run (46b) ns/op: 32.3556 Single filter ns/op: 83.2239 Random filter ns/op: 370.676 ---------------------------- Outside queries... Dry run (25d) ns/op: 32.2265 Single filter ns/op: 93.5651 Random filter ns/op: 408.393 Average FP rate %: 1.13993 ---------------------------- Done. (For more info, run with -legend or -help.) ``` With the new code: ``` $ ./filter_bench -quick WARNING: Assertions are enabled; benchmarks unnecessarily slow Building... Build avg ns/key: 25.4285 Number of filters: 16669 Total memory (MB): 200.009 Bits/key actual: 10.0647 ---------------------------- Inside queries... Dry run (46b) ns/op: 31.0594 Single filter ns/op: 43.8974 Random filter ns/op: 226.075 ---------------------------- Outside queries... Dry run (25d) ns/op: 31.0295 Single filter ns/op: 50.3824 Random filter ns/op: 226.805 Average FP rate %: 1.13993 ---------------------------- Done. (For more info, run with -legend or -help.) $ ./filter_bench -quick -use_full_block_reader WARNING: Assertions are enabled; benchmarks unnecessarily slow Building... Build avg ns/key: 26.5308 Number of filters: 16669 Total memory (MB): 200.009 Bits/key actual: 10.0647 ---------------------------- Inside queries... Dry run (46b) ns/op: 33.2968 Single filter ns/op: 58.6163 Random filter ns/op: 291.434 ---------------------------- Outside queries... Dry run (25d) ns/op: 32.1839 Single filter ns/op: 66.9039 Random filter ns/op: 292.828 Average FP rate %: 1.13993 ---------------------------- Done. (For more info, run with -legend or -help.) ``` Differential Revision: D17991712 Pulled By: ltamasi fbshipit-source-id: 7ea205550217bfaaa1d5158ebd658e5832e60f29		5 years ago
..
block.cc	Add an option to put first key of each sst block in the index (#5289)	6 years ago
block.h	Add class comment for Block	5 years ago
block_based_filter_block.cc	Fix regression affecting partitioned indexes/filters when cache_index_and_filter_blocks is false (#5705)	5 years ago
block_based_filter_block.h	Use delete to disable automatic generated methods. (#5009)	5 years ago
block_based_filter_block_test.cc	filter_bench - a prelim tool for SST filter benchmarking (#5825)	5 years ago
block_based_table_builder.cc	Add support for timestamp in Get/Put (#5079)	6 years ago
block_based_table_builder.h	Use delete to disable automatic generated methods. (#5009)	5 years ago
block_based_table_factory.cc	Apply formatter to recent 200+ commits. (#5830)	5 years ago
block_based_table_factory.h	Organizing rocksdb/table directory by format	6 years ago
block_based_table_reader.cc	Store the filter bits reader alongside the filter block contents (#5936)	5 years ago
block_based_table_reader.h	Revert "Merging iterator to avoid child iterator reseek for some cases (#5286)" (#5871)	5 years ago
block_builder.cc	Organizing rocksdb/table directory by format	6 years ago
block_builder.h	Organizing rocksdb/table directory by format	6 years ago
block_prefix_index.cc	Move some memory related files from util/ to memory/ (#5382)	6 years ago
block_prefix_index.h	Organizing rocksdb/table directory by format	6 years ago
block_test.cc	upgrade gtest 1.7.0 => 1.8.1 for json result writing	5 years ago
block_type.h	Make the 'block read count' performance counters consistent (#5484)	6 years ago
cachable_entry.h	Move the filter readers out of the block cache (#5504)	6 years ago
data_block_footer.cc	Organizing rocksdb/table directory by format	6 years ago
data_block_footer.h	Organizing rocksdb/table directory by format	6 years ago
data_block_hash_index.cc	Organizing rocksdb/table directory by format	6 years ago
data_block_hash_index.h	Organizing rocksdb/table directory by format	6 years ago
data_block_hash_index_test.cc	New API to get all merge operands for a Key (#5604)	5 years ago
filter_block.h	Apply formatter to recent 200+ commits. (#5830)	5 years ago
filter_block_reader_common.cc	Store the filter bits reader alongside the filter block contents (#5936)	5 years ago
filter_block_reader_common.h	Fix regression affecting partitioned indexes/filters when cache_index_and_filter_blocks is false (#5705)	5 years ago
flush_block_policy.cc	Organizing rocksdb/table directory by format	6 years ago
flush_block_policy.h	Organizing rocksdb/table directory by format	6 years ago
full_filter_block.cc	Store the filter bits reader alongside the filter block contents (#5936)	5 years ago
full_filter_block.h	Store the filter bits reader alongside the filter block contents (#5936)	5 years ago
full_filter_block_test.cc	Store the filter bits reader alongside the filter block contents (#5936)	5 years ago
index_builder.cc	Add an option to put first key of each sst block in the index (#5289)	6 years ago
index_builder.h	Add an option to put first key of each sst block in the index (#5289)	6 years ago
mock_block_based_table.h	filter_bench - a prelim tool for SST filter benchmarking (#5825)	5 years ago
parsed_full_filter_block.cc	Store the filter bits reader alongside the filter block contents (#5936)	5 years ago
parsed_full_filter_block.h	Store the filter bits reader alongside the filter block contents (#5936)	5 years ago
partitioned_filter_block.cc	Store the filter bits reader alongside the filter block contents (#5936)	5 years ago
partitioned_filter_block.h	Store the filter bits reader alongside the filter block contents (#5936)	5 years ago
partitioned_filter_block_test.cc	Store the filter bits reader alongside the filter block contents (#5936)	5 years ago
uncompression_dict_reader.cc	Apply formatter to recent 200+ commits. (#5830)	5 years ago
uncompression_dict_reader.h	Revert to storing UncompressionDicts in the cache (#5645)	5 years ago