fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Andrew Kryczka 6c40806e51 Digest ZSTD compression dictionary once per SST file (#4251) ... Summary: In RocksDB, for a given SST file, all data blocks are compressed with the same dictionary. When we compress a block using the dictionary's raw bytes, the compression library first has to digest the dictionary to get it into a usable form. This digestion work is redundant and ideally should be done once per file. ZSTD offers APIs for the caller to create and reuse a digested dictionary object (`ZSTD_CDict`). In this PR, we call `ZSTD_createCDict` once per file to digest the raw bytes. Then we use `ZSTD_compress_usingCDict` to compress each data block using the pre-digested dictionary. Once the file's created `ZSTD_freeCDict` releases the resources held by the digested dictionary. There are a couple other changes included in this PR: - Changed the parameter object for (un)compression functions from `CompressionContext`/`UncompressionContext` to `CompressionInfo`/`UncompressionInfo`. This avoids the previous pattern, where `CompressionContext`/`UncompressionContext` had to be mutated before calling a (un)compression function depending on whether dictionary should be used. I felt that mutation was error-prone so eliminated it. - Added support for digested uncompression dictionaries (`ZSTD_DDict`) as well. However, this PR does not support reusing them across uncompression calls for the same file. That work is deferred to a later PR when we will store the `ZSTD_DDict` objects in block cache. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4251 Differential Revision: D9257078 Pulled By: ajkr fbshipit-source-id: 21b8cb6bbdd48e459f1c62343780ab66c0a64438		6 years ago
..
adaptive_table_factory.cc	Comment out unused variables	7 years ago
adaptive_table_factory.h	Comment out unused variables	7 years ago
block.cc	DataBlockHashIndex: avoiding expensive iiter->Next when handling hash kNoEntry (#4296)	6 years ago
block.h	Improve point-lookup performance using a data block hash index (#4174)	6 years ago
block_based_filter_block.cc	PrefixMayMatch: remove unnecessary check for prefix_extractor_ (#4067)	7 years ago
block_based_filter_block.h	Move prefix_extractor to MutableCFOptions	7 years ago
block_based_filter_block_test.cc	Move prefix_extractor to MutableCFOptions	7 years ago
block_based_table_builder.cc	Digest ZSTD compression dictionary once per SST file (#4251)	6 years ago
block_based_table_builder.h	Digest ZSTD compression dictionary once per SST file (#4251)	6 years ago
block_based_table_factory.cc	Add db_bench options of data block hash index (#4281)	6 years ago
block_based_table_factory.h	Improve point-lookup performance using a data block hash index (#4174)	6 years ago
block_based_table_reader.cc	Digest ZSTD compression dictionary once per SST file (#4251)	6 years ago
block_based_table_reader.h	Index value delta encoding (#3983)	6 years ago
block_builder.cc	DataBlockHashIndex: Remove the division from EstimateSize() (#4293)	6 years ago
block_builder.h	Add path to WritableFileWriter. (#4039)	6 years ago
block_fetcher.cc	Digest ZSTD compression dictionary once per SST file (#4251)	6 years ago
block_fetcher.h	Pin mmap files in ReadOnlyDB (#4053)	7 years ago
block_prefix_index.cc	Change RocksDB License	8 years ago
block_prefix_index.h	Change RocksDB License	8 years ago
block_test.cc	Add path to WritableFileWriter. (#4039)	6 years ago
bloom_block.cc	Change RocksDB License	8 years ago
bloom_block.h	Change RocksDB License	8 years ago
cleanable_test.cc	Change RocksDB License	8 years ago
cuckoo_table_builder.cc	Enable MSVC W4 with a few exceptions. Fix warnings and bugs	7 years ago
cuckoo_table_builder.h	Change RocksDB License	8 years ago
cuckoo_table_builder_test.cc	Add path to WritableFileWriter. (#4039)	6 years ago
cuckoo_table_factory.cc	Comment out unused variables	7 years ago
cuckoo_table_factory.h	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
cuckoo_table_reader.cc	Index value delta encoding (#3983)	6 years ago
cuckoo_table_reader.h	Index value delta encoding (#3983)	6 years ago
cuckoo_table_reader_test.cc	Add path to WritableFileWriter. (#4039)	6 years ago
data_block_footer.cc	Add db_bench options of data block hash index (#4281)	6 years ago
data_block_footer.h	Add db_bench options of data block hash index (#4281)	6 years ago
data_block_hash_index.cc	DataBlockHashIndex: Remove the division from EstimateSize() (#4293)	6 years ago
data_block_hash_index.h	DataBlockHashIndex: Remove the division from EstimateSize() (#4293)	6 years ago
data_block_hash_index_test.cc	Add path to WritableFileWriter. (#4039)	6 years ago
filter_block.h	use user_key and iterate_upper_bound to determine compatibility of bloom filters (#3899)	7 years ago
flush_block_policy.cc	Align SST file data blocks to avoid spanning multiple pages	7 years ago
format.cc	Digest ZSTD compression dictionary once per SST file (#4251)	6 years ago
format.h	Digest ZSTD compression dictionary once per SST file (#4251)	6 years ago
full_filter_bits_builder.h	Skip duplicate bloom keys when whole_key and prefix are mixed	7 years ago
full_filter_block.cc	Charging block cache more accurately (#4073)	7 years ago
full_filter_block.h	Charging block cache more accurately (#4073)	7 years ago
full_filter_block_test.cc	Move prefix_extractor to MutableCFOptions	7 years ago
get_context.cc	#3865 followup for fix performance degression introduced by switching order of operands (#4284)	6 years ago
get_context.h	Avoid unnecessary big for-loop when reporting ticker stats stored in GetContext (#3490)	7 years ago
index_builder.cc	Add path to WritableFileWriter. (#4039)	6 years ago
index_builder.h	Add path to WritableFileWriter. (#4039)	6 years ago
internal_iterator.h	Index value delta encoding (#3983)	6 years ago
iter_heap.h	Make InternalKeyComparator final and directly use it in merging iterator	7 years ago
iterator.cc	Index value delta encoding (#3983)	6 years ago
iterator_wrapper.h	Index value delta encoding (#3983)	6 years ago
merger_test.cc	Make InternalKeyComparator final and directly use it in merging iterator	7 years ago
merging_iterator.cc	Index value delta encoding (#3983)	6 years ago
merging_iterator.h	Index value delta encoding (#3983)	6 years ago
meta_blocks.cc	Index value delta encoding (#3983)	6 years ago
meta_blocks.h	Index value delta encoding (#3983)	6 years ago
mock_table.cc	Add path to WritableFileWriter. (#4039)	6 years ago
mock_table.h	Make BlockBasedTableIterator compaction-aware (#4048)	7 years ago
partitioned_filter_block.cc	Index value delta encoding (#3983)	6 years ago
partitioned_filter_block.h	Index value delta encoding (#3983)	6 years ago
partitioned_filter_block_test.cc	Index value delta encoding (#3983)	6 years ago
persistent_cache_helper.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
persistent_cache_helper.h	Change RocksDB License	8 years ago
persistent_cache_options.h	Change RocksDB License	8 years ago
plain_table_builder.cc	Move prefix_extractor to MutableCFOptions	7 years ago
plain_table_builder.h	Move prefix_extractor to MutableCFOptions	7 years ago
plain_table_factory.cc	Move prefix_extractor to MutableCFOptions	7 years ago
plain_table_factory.h	Comment out unused variables	7 years ago
plain_table_index.cc	Change RocksDB License	8 years ago
plain_table_index.h	Move prefix_extractor to MutableCFOptions	7 years ago
plain_table_key_coding.cc	Comment out unused variables	7 years ago
plain_table_key_coding.h	Change RocksDB License	8 years ago
plain_table_reader.cc	Make BlockBasedTableIterator compaction-aware (#4048)	7 years ago
plain_table_reader.h	Index value delta encoding (#3983)	6 years ago
scoped_arena_iterator.h	Change RocksDB License	8 years ago
sst_file_writer.cc	Add path to WritableFileWriter. (#4039)	6 years ago
sst_file_writer_collectors.h	Comment out unused variables	7 years ago
table_builder.h	Remove random writes from SST file ingestion (#4172)	7 years ago
table_properties.cc	Index value delta encoding (#3983)	6 years ago
table_properties_internal.h	Index value delta encoding (#3983)	6 years ago
table_reader.h	Index value delta encoding (#3983)	6 years ago
table_reader_bench.cc	Add path to WritableFileWriter. (#4039)	6 years ago
table_test.cc	Add path to WritableFileWriter. (#4039)	6 years ago
two_level_iterator.cc	Index value delta encoding (#3983)	6 years ago
two_level_iterator.h	Index value delta encoding (#3983)	6 years ago