fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Andrew Kryczka 62f70f6d14 Reduce scope of compression dictionary to single SST (#4952) ... Summary: Our previous approach was to train one compression dictionary per compaction, using the first output SST to train a dictionary, and then applying it on subsequent SSTs in the same compaction. While this was great for minimizing CPU/memory/I/O overhead, it did not achieve good compression ratios in practice. In our most promising potential use case, moderate reductions in a dictionary's scope make a major difference on compression ratio. So, this PR changes compression dictionary to be scoped per-SST. It accepts the tradeoff during table building to use more memory and CPU. Important changes include: - The `BlockBasedTableBuilder` has a new state when dictionary compression is in-use: `kBuffered`. In that state it accumulates uncompressed data in-memory whenever `Add` is called. - After accumulating target file size bytes or calling `BlockBasedTableBuilder::Finish`, a `BlockBasedTableBuilder` moves to the `kUnbuffered` state. The transition (`EnterUnbuffered()`) involves sampling the buffered data, training a dictionary, and compressing/writing out all buffered data. In the `kUnbuffered` state, a `BlockBasedTableBuilder` behaves the same as before -- blocks are compressed/written out as soon as they fill up. - Samples are now whole uncompressed data blocks, except the final sample may be a partial data block so we don't breach the user's configured `max_dict_bytes` or `zstd_max_train_bytes`. The dictionary trainer is supposed to work better when we pass it real units of compression. Previously we were passing 64-byte KV samples which was not realistic. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4952 Differential Revision: D13967980 Pulled By: ajkr fbshipit-source-id: 82bea6f7537e1529c7a1a4cdee84585f5949300f		6 years ago
..
adaptive_table_factory.cc	Update all unique/shared_ptr instances to be qualified with namespace std (#4638)	6 years ago
adaptive_table_factory.h	Update all unique/shared_ptr instances to be qualified with namespace std (#4638)	6 years ago
block.cc	Checksum properties block for block-based table (#4956)	6 years ago
block.h	Checksum properties block for block-based table (#4956)	6 years ago
block_based_filter_block.cc	PrefixMayMatch: remove unnecessary check for prefix_extractor_ (#4067)	7 years ago
block_based_filter_block.h	Move prefix_extractor to MutableCFOptions	7 years ago
block_based_filter_block_test.cc	Remove two variables from BlockContents class and don't use class Block for compressed block (#4650)	6 years ago
block_based_table_builder.cc	Reduce scope of compression dictionary to single SST (#4952)	6 years ago
block_based_table_builder.h	Reduce scope of compression dictionary to single SST (#4952)	6 years ago
block_based_table_factory.cc	Reduce scope of compression dictionary to single SST (#4952)	6 years ago
block_based_table_factory.h	Revert "Move MemoryAllocator option from Cache to BlockBasedTableOpti… (#4697)	6 years ago
block_based_table_reader.cc	Checksum properties block for block-based table (#4956)	6 years ago
block_based_table_reader.h	Cache dictionary used for decompressing data blocks (#4881)	6 years ago
block_builder.cc	DataBlockHashIndex: Remove the division from EstimateSize() (#4293)	6 years ago
block_builder.h	Add path to WritableFileWriter. (#4039)	6 years ago
block_fetcher.cc	Cache dictionary used for decompressing data blocks (#4881)	6 years ago
block_fetcher.h	Cache dictionary used for decompressing data blocks (#4881)	6 years ago
block_prefix_index.cc	Change RocksDB License	8 years ago
block_prefix_index.h	Change RocksDB License	8 years ago
block_test.cc	Remove two variables from BlockContents class and don't use class Block for compressed block (#4650)	6 years ago
bloom_block.cc	Change RocksDB License	8 years ago
bloom_block.h	Disallow customized hash function in DynamicBloom (#4915)	6 years ago
cleanable_test.cc	Change RocksDB License	8 years ago
cuckoo_table_builder.cc	Promote rocksdb.{deleted.keys,merge.operands} to main table properties (#4594)	6 years ago
cuckoo_table_builder.h	Change RocksDB License	8 years ago
cuckoo_table_builder_test.cc	Update all unique/shared_ptr instances to be qualified with namespace std (#4638)	6 years ago
cuckoo_table_factory.cc	Update all unique/shared_ptr instances to be qualified with namespace std (#4638)	6 years ago
cuckoo_table_factory.h	Update all unique/shared_ptr instances to be qualified with namespace std (#4638)	6 years ago
cuckoo_table_reader.cc	Support pragma once in all header files and cleanup some warnings (#4339)	6 years ago
cuckoo_table_reader.h	Index value delta encoding (#3983)	6 years ago
cuckoo_table_reader_test.cc	Update all unique/shared_ptr instances to be qualified with namespace std (#4638)	6 years ago
data_block_footer.cc	Add db_bench options of data block hash index (#4281)	6 years ago
data_block_footer.h	Add db_bench options of data block hash index (#4281)	6 years ago
data_block_hash_index.cc	DataBlockHashIndex: Remove the division from EstimateSize() (#4293)	6 years ago
data_block_hash_index.h	DataBlockHashIndex: Remove the division from EstimateSize() (#4293)	6 years ago
data_block_hash_index_test.cc	Reduce scope of compression dictionary to single SST (#4952)	6 years ago
filter_block.h	use user_key and iterate_upper_bound to determine compatibility of bloom filters (#3899)	7 years ago
flush_block_policy.cc	Align SST file data blocks to avoid spanning multiple pages	7 years ago
format.cc	Digest ZSTD compression dictionary once when writing SST file (#4849)	6 years ago
format.h	Digest ZSTD compression dictionary once when writing SST file (#4849)	6 years ago
full_filter_bits_builder.h	Skip duplicate bloom keys when whole_key and prefix are mixed	7 years ago
full_filter_block.cc	Charging block cache more accurately (#4073)	7 years ago
full_filter_block.h	Charging block cache more accurately (#4073)	7 years ago
full_filter_block_test.cc	Move prefix_extractor to MutableCFOptions	7 years ago
get_context.cc	PlainTable should avoid copying Get() results from immortal source. (#4924)	6 years ago
get_context.h	Cache dictionary used for decompressing data blocks (#4881)	6 years ago
index_builder.cc	Add path to WritableFileWriter. (#4039)	6 years ago
index_builder.h	Add path to WritableFileWriter. (#4039)	6 years ago
internal_iterator.h	Index value delta encoding (#3983)	6 years ago
iter_heap.h	Make InternalKeyComparator final and directly use it in merging iterator	7 years ago
iterator.cc	fix typo in error message, twice (#4457)	6 years ago
iterator_wrapper.h	Index value delta encoding (#3983)	6 years ago
merger_test.cc	Make InternalKeyComparator final and directly use it in merging iterator	7 years ago
merging_iterator.cc	Index value delta encoding (#3983)	6 years ago
merging_iterator.h	Index value delta encoding (#3983)	6 years ago
meta_blocks.cc	Checksum properties block for block-based table (#4956)	6 years ago
meta_blocks.h	Checksum properties block for block-based table (#4956)	6 years ago
mock_table.cc	Update all unique/shared_ptr instances to be qualified with namespace std (#4638)	6 years ago
mock_table.h	Update all unique/shared_ptr instances to be qualified with namespace std (#4638)	6 years ago
partitioned_filter_block.cc	Fix bug in partition filters with format_version=4 (#4381)	6 years ago
partitioned_filter_block.h	Fix bug in partition filters with format_version=4 (#4381)	6 years ago
partitioned_filter_block_test.cc	Remove two variables from BlockContents class and don't use class Block for compressed block (#4650)	6 years ago
persistent_cache_helper.cc	Remove two variables from BlockContents class and don't use class Block for compressed block (#4650)	6 years ago
persistent_cache_helper.h	Change RocksDB License	8 years ago
persistent_cache_options.h	Change RocksDB License	8 years ago
plain_table_builder.cc	Remove PlainTable's feature store_index_in_file (#4914)	6 years ago
plain_table_builder.h	Remove PlainTable's feature store_index_in_file (#4914)	6 years ago
plain_table_factory.cc	Remove cuckoo hash memtable (#4953)	6 years ago
plain_table_factory.h	Remove PlainTable's feature store_index_in_file (#4914)	6 years ago
plain_table_index.cc	Change RocksDB License	8 years ago
plain_table_index.h	Move prefix_extractor to MutableCFOptions	7 years ago
plain_table_key_coding.cc	Comment out unused variables	7 years ago
plain_table_key_coding.h	Update all unique/shared_ptr instances to be qualified with namespace std (#4638)	6 years ago
plain_table_reader.cc	Remove PlainTable's feature store_index_in_file (#4914)	6 years ago
plain_table_reader.h	PlainTable should avoid copying Get() results from immortal source. (#4924)	6 years ago
scoped_arena_iterator.h	Change RocksDB License	8 years ago
sst_file_reader.cc	Get `CompactionJobInfo` from CompactFiles	6 years ago
sst_file_reader_test.cc	Get `CompactionJobInfo` from CompactFiles	6 years ago
sst_file_writer.cc	Reduce scope of compression dictionary to single SST (#4952)	6 years ago
sst_file_writer_collectors.h	Comment out unused variables	7 years ago
table_builder.h	Reduce scope of compression dictionary to single SST (#4952)	6 years ago
table_properties.cc	Promote rocksdb.{deleted.keys,merge.operands} to main table properties (#4594)	6 years ago
table_properties_internal.h	Index value delta encoding (#3983)	6 years ago
table_reader.h	Clean up FragmentedRangeTombstoneList (#4692)	6 years ago
table_reader_bench.cc	Reduce scope of compression dictionary to single SST (#4952)	6 years ago
table_test.cc	Reduce scope of compression dictionary to single SST (#4952)	6 years ago
two_level_iterator.cc	Index value delta encoding (#3983)	6 years ago
two_level_iterator.h	Index value delta encoding (#3983)	6 years ago