fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Dhruba Borthakur 321dfdc3ae Allow having different compression algorithms on different levels. ... Summary: The leveldb API is enhanced to support different compression algorithms at different levels. This adds the option min_level_to_compress to db_bench that specifies the minimum level for which compression should be done when compression is enabled. This can be used to disable compression for levels 0 and 1 which are likely to suffer from stalls because of the CPU load for memtable flushes and (L0,L1) compaction. Level 0 is special as it gets frequent memtable flushes. Level 1 is special as it frequently gets all:all file compactions between it and level 0. But all other levels could be the same. For any level N where N > 1, the rate of sequential IO for that level should be the same. The last level is the exception because it might not be full and because files from it are not read to compact with the next larger level. The same amount of time will be spent doing compaction at any level N excluding N=0, 1 or the last level. By this standard all of those levels should use the same compression. The difference is that the loss (using more disk space) from a faster compression algorithm is less significant for N=2 than for N=3. So we might be willing to trade disk space for faster write rates with no compression for L0 and L1, snappy for L2, zlib for L3. Using a faster compression algorithm for the mid levels also allows us to reclaim some cpu without trading off much loss in disk space overhead. Also note that little is to be gained by compressing levels 0 and 1. For a 4-level tree they account for 10% of the data. For a 5-level tree they account for 1% of the data. With compression enabled: * memtable flush rate is ~18MB/second * (L0,L1) compaction rate is ~30MB/second With compression enabled but min_level_to_compress=2 * memtable flush rate is ~320MB/second * (L0,L1) compaction rate is ~560MB/second This practicaly takes the same code from https://reviews.facebook.net/D6225 but makes the leveldb api more general purpose with a few additional lines of code. Test Plan: make check Differential Revision: https://reviews.facebook.net/D6261		12 years ago
..
builder.cc	Allow having different compression algorithms on different levels.	12 years ago
builder.h	A number of fixes:	13 years ago
c.cc	put log in a seperate dir	12 years ago
c_test.c	merge 1.5	12 years ago
corruption_test.cc	Make some variables configurable for each db instance	13 years ago
db_bench.cc	Allow having different compression algorithms on different levels.	12 years ago
db_filesnapshot.cc	The BackupAPI should also list the length of the manifest file.	12 years ago
db_impl.cc	Allow having different compression algorithms on different levels.	12 years ago
db_impl.h	Adds DB::GetNextCompaction and then uses that for rate limiting db_bench	12 years ago
db_iter.cc	A number of fixes:	13 years ago
db_iter.h	A number of fixes:	13 years ago
db_statistics.h	Fix table-cache size bug, gather table-cache statistics and prevent readahead done by fs. Summary:	13 years ago
db_stats_logger.cc	remove boost	12 years ago
db_test.cc	Allow having different compression algorithms on different levels.	12 years ago
dbformat.cc	Added bloom filter support.	13 years ago
dbformat.h	Make some variables configurable for each db instance	13 years ago
dbformat_test.cc	A number of fixes:	13 years ago
filename.cc	put log in a seperate dir	12 years ago
filename.h	put log in a seperate dir	12 years ago
filename_test.cc	A number of fixes:	13 years ago
log_format.h	A number of fixes:	13 years ago
log_reader.cc	A number of fixes:	13 years ago
log_reader.h	A number of fixes:	13 years ago
log_test.cc	A number of fixes:	13 years ago
log_writer.cc	A number of fixes:	13 years ago
log_writer.h	A number of fixes:	13 years ago
memtable.cc	A number of fixes:	13 years ago
memtable.h	A number of fixes:	13 years ago
repair.cc	Make some variables configurable for each db instance	13 years ago
skiplist.h	skiplist: optimize for sequential insert pattern	13 years ago
skiplist_test.cc	A number of fixes:	13 years ago
snapshot.h	A number of fixes:	13 years ago
table_cache.cc	Trigger read compaction only if seeks to storage are incurred.	12 years ago
table_cache.h	Trigger read compaction only if seeks to storage are incurred.	12 years ago
version_edit.cc	Clean up compiler warnings generated by -Wall option.	12 years ago
version_edit.h	Make some variables configurable for each db instance	13 years ago
version_edit_test.cc	Make some variables configurable for each db instance	13 years ago
version_set.cc	add "seek_compaction" to log for better debug Summary:	12 years ago
version_set.h	Adds DB::GetNextCompaction and then uses that for rate limiting db_bench	12 years ago
version_set_test.cc	A number of fixes:	13 years ago
write_batch.cc	added group commit; drastically speeds up mult-threaded synchronous write workloads	13 years ago
write_batch_internal.h	added group commit; drastically speeds up mult-threaded synchronous write workloads	13 years ago
write_batch_test.cc	added group commit; drastically speeds up mult-threaded synchronous write workloads	13 years ago