fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Andrew Kryczka cc01985db0 Introduce bottom-pri thread pool for large universal compactions ... Summary: When we had a single thread pool for compactions, a thread could be busy for a long time (minutes) executing a compaction involving the bottom level. In multi-instance setups, the entire thread pool could be consumed by such bottom-level compactions. Then, top-level compactions (e.g., a few L0 files) would be blocked for a long time ("head-of-line blocking"). Such top-level compactions are critical to prevent compaction stalls as they can quickly reduce number of L0 files / sorted runs. This diff introduces a bottom-priority queue for universal compactions including the bottom level. This alleviates the head-of-line blocking situation for fast, top-level compactions. - Added `Env::Priority::BOTTOM` thread pool. This feature is only enabled if user explicitly configures it to have a positive number of threads. - Changed `ThreadPoolImpl`'s default thread limit from one to zero. This change is invisible to users as we call `IncBackgroundThreadsIfNeeded` on the low-pri/high-pri pools during `DB::Open` with values of at least one. It is necessary, though, for bottom-pri to start with zero threads so the feature is disabled by default. - Separated `ManualCompaction` into two parts in `PrepickedCompaction`. `PrepickedCompaction` is used for any compaction that's picked outside of its execution thread, either manual or automatic. - Forward universal compactions involving last level to the bottom pool (worker thread's entry point is `BGWorkBottomCompaction`). - Track `bg_bottom_compaction_scheduled_` so we can wait for bottom-level compactions to finish. We don't count them against the background jobs limits. So users of this feature will get an extra compaction for free. Closes https://github.com/facebook/rocksdb/pull/2580 Differential Revision: D5422916 Pulled By: ajkr fbshipit-source-id: a74bd11f1ea4933df3739b16808bb21fcd512333		7 years ago
..
builder.cc	Revert "comment out unused parameters"	8 years ago
builder.h	Change RocksDB License	8 years ago
c.cc	Added db paths to c	8 years ago
c_test.c	Added db paths to c	8 years ago
column_family.cc	Change RocksDB License	8 years ago
column_family.h	Change RocksDB License	8 years ago
column_family_test.cc	Revert "comment out unused parameters"	8 years ago
compact_files_test.cc	Revert "comment out unused parameters"	8 years ago
compacted_db_impl.cc	Change RocksDB License	8 years ago
compacted_db_impl.h	Revert "comment out unused parameters"	8 years ago
compaction.cc	Fix universal compaction bug	8 years ago
compaction.h	Change RocksDB License	8 years ago
compaction_iteration_stats.h	Change RocksDB License	8 years ago
compaction_iterator.cc	Revert "comment out unused parameters"	8 years ago
compaction_iterator.h	Revert "comment out unused parameters"	8 years ago
compaction_iterator_test.cc	Revert "comment out unused parameters"	8 years ago
compaction_job.cc	Fix compaction div by zero logging	8 years ago
compaction_job.h	Change RocksDB License	8 years ago
compaction_job_stats_test.cc	Revert "comment out unused parameters"	8 years ago
compaction_job_test.cc	Change RocksDB License	8 years ago
compaction_picker.cc	Fix caching of compaction picker's next index	8 years ago
compaction_picker.h	Revert "comment out unused parameters"	8 years ago
compaction_picker_test.cc	Fix FIFO compaction picker test	8 years ago
compaction_picker_universal.cc	Change RocksDB License	8 years ago
compaction_picker_universal.h	Change RocksDB License	8 years ago
comparator_db_test.cc	Revert "comment out unused parameters"	8 years ago
convenience.cc	Replace dynamic_cast<>	8 years ago
corruption_test.cc	CodeMod: Prefer ADD_FAILURE() over EXPECT_TRUE(false), et cetera	8 years ago
cuckoo_table_db_test.cc	Change RocksDB License	8 years ago
db_basic_test.cc	Build fewer tests in Travis platform_dependent tests	8 years ago
db_block_cache_test.cc	Revert "comment out unused parameters"	8 years ago
db_bloom_filter_test.cc	Revert "comment out unused parameters"	8 years ago
db_compaction_filter_test.cc	Revert "comment out unused parameters"	8 years ago
db_compaction_test.cc	Revert "comment out unused parameters"	8 years ago
db_dynamic_level_test.cc	Revert "comment out unused parameters"	8 years ago
db_encryption_test.cc	Change RocksDB License	8 years ago
db_filesnapshot.cc	Remove some left-over BSD headers	8 years ago
db_flush_test.cc	Revert "comment out unused parameters"	8 years ago
db_impl.cc	Introduce bottom-pri thread pool for large universal compactions	7 years ago
db_impl.h	Introduce bottom-pri thread pool for large universal compactions	7 years ago
db_impl_compaction_flush.cc	Introduce bottom-pri thread pool for large universal compactions	7 years ago
db_impl_debug.cc	Introduce bottom-pri thread pool for large universal compactions	7 years ago
db_impl_experimental.cc	Change RocksDB License	8 years ago
db_impl_files.cc	move TableCache::EraseHandle outside of db mutex	8 years ago
db_impl_open.cc	Change RocksDB License	8 years ago
db_impl_readonly.cc	Add Iterator::Refresh()	8 years ago
db_impl_readonly.h	Revert "comment out unused parameters"	8 years ago
db_impl_write.cc	Remove the orphan assert on !need_log_sync	8 years ago
db_info_dumper.cc	Change RocksDB License	8 years ago
db_info_dumper.h	Change RocksDB License	8 years ago
db_inplace_update_test.cc	Change RocksDB License	8 years ago
db_io_failure_test.cc	Fix LITE unit tests	8 years ago
db_iter.cc	Add Iterator::Refresh()	8 years ago
db_iter.h	Add Iterator::Refresh()	8 years ago
db_iter_test.cc	Add Iterator::Refresh()	8 years ago
db_iterator_test.cc	Add Iterator::Refresh()	8 years ago
db_log_iter_test.cc	Change RocksDB License	8 years ago
db_memtable_test.cc	Revert "comment out unused parameters"	8 years ago
db_merge_operator_test.cc	Change RocksDB License	8 years ago
db_options_test.cc	Change RocksDB License	8 years ago
db_properties_test.cc	Revert "comment out unused parameters"	8 years ago
db_range_del_test.cc	Change RocksDB License	8 years ago
db_sst_test.cc	Revert "comment out unused parameters"	8 years ago
db_statistics_test.cc	Change RocksDB License	8 years ago
db_table_properties_test.cc	Change RocksDB License	8 years ago
db_tailing_iter_test.cc	Revert "comment out unused parameters"	8 years ago
db_test.cc	Fix FIFO Compaction with TTL tests	8 years ago
db_test2.cc	Fix LITE unit tests	8 years ago
db_test_util.cc	Fix LITE unit tests	8 years ago
db_test_util.h	Revert "comment out unused parameters"	8 years ago
db_universal_compaction_test.cc	Introduce bottom-pri thread pool for large universal compactions	7 years ago
db_wal_test.cc	Change RocksDB License	8 years ago
db_write_test.cc	Change RocksDB License	8 years ago
dbformat.cc	Change RocksDB License	8 years ago
dbformat.h	Replace dynamic_cast<>	8 years ago
dbformat_test.cc	Change RocksDB License	8 years ago
deletefile_test.cc	Revert "comment out unused parameters"	8 years ago
event_helpers.cc	Change RocksDB License	8 years ago
event_helpers.h	Change RocksDB License	8 years ago
experimental.cc	Replace dynamic_cast<>	8 years ago
external_sst_file_basic_test.cc	Change RocksDB License	8 years ago
external_sst_file_ingestion_job.cc	Change RocksDB License	8 years ago
external_sst_file_ingestion_job.h	Change RocksDB License	8 years ago
external_sst_file_test.cc	Revert "comment out unused parameters"	8 years ago
fault_injection_test.cc	Revert "comment out unused parameters"	8 years ago
file_indexer.cc	Change RocksDB License	8 years ago
file_indexer.h	Change RocksDB License	8 years ago
file_indexer_test.cc	Revert "comment out unused parameters"	8 years ago
filename_test.cc	Change RocksDB License	8 years ago
flush_job.cc	Change RocksDB License	8 years ago
flush_job.h	Change RocksDB License	8 years ago
flush_job_test.cc	Change RocksDB License	8 years ago
flush_scheduler.cc	Change RocksDB License	8 years ago
flush_scheduler.h	Change RocksDB License	8 years ago
forward_iterator.cc	Revert "comment out unused parameters"	8 years ago
forward_iterator.h	Revert "comment out unused parameters"	8 years ago
forward_iterator_bench.cc	Change RocksDB License	8 years ago
internal_stats.cc	Revert "comment out unused parameters"	8 years ago
internal_stats.h	Change RocksDB License	8 years ago
job_context.h	Change RocksDB License	8 years ago
listener_test.cc	Revert "comment out unused parameters"	8 years ago
log_format.h	Change RocksDB License	8 years ago
log_reader.cc	Change RocksDB License	8 years ago
log_reader.h	Change RocksDB License	8 years ago
log_test.cc	Change RocksDB License	8 years ago
log_writer.cc	Change RocksDB License	8 years ago
log_writer.h	Change RocksDB License	8 years ago
malloc_stats.cc	Revert "comment out unused parameters"	8 years ago
malloc_stats.h	Change RocksDB License	8 years ago
managed_iterator.cc	Change RocksDB License	8 years ago
managed_iterator.h	Change RocksDB License	8 years ago
manual_compaction_test.cc	Revert "comment out unused parameters"	8 years ago
memtable.cc	Change RocksDB License	8 years ago
memtable.h	Change RocksDB License	8 years ago
memtable_list.cc	Revert "comment out unused parameters"	8 years ago
memtable_list.h	Change RocksDB License	8 years ago
memtable_list_test.cc	Change RocksDB License	8 years ago
merge_context.h	Change RocksDB License	8 years ago
merge_helper.cc	Change RocksDB License	8 years ago
merge_helper.h	Change RocksDB License	8 years ago
merge_helper_test.cc	Change RocksDB License	8 years ago
merge_operator.cc	Change RocksDB License	8 years ago
merge_test.cc	Revert "comment out unused parameters"	8 years ago
options_file_test.cc	Change RocksDB License	8 years ago
perf_context_test.cc	Change RocksDB License	8 years ago
pinned_iterators_manager.h	Change RocksDB License	8 years ago
plain_table_db_test.cc	Revert "comment out unused parameters"	8 years ago
prefix_test.cc	Revert "comment out unused parameters"	8 years ago
range_del_aggregator.cc	Change RocksDB License	8 years ago
range_del_aggregator.h	Change RocksDB License	8 years ago
range_del_aggregator_test.cc	Change RocksDB License	8 years ago
repair.cc	Change RocksDB License	8 years ago
repair_test.cc	Change RocksDB License	8 years ago
snapshot_impl.cc	Change RocksDB License	8 years ago
snapshot_impl.h	Change RocksDB License	8 years ago
table_cache.cc	fix db get/write stats	8 years ago
table_cache.h	Change RocksDB License	8 years ago
table_properties_collector.cc	Revert "comment out unused parameters"	8 years ago
table_properties_collector.h	Revert "comment out unused parameters"	8 years ago
table_properties_collector_test.cc	Revert "comment out unused parameters"	8 years ago
transaction_log_impl.cc	Change RocksDB License	8 years ago
transaction_log_impl.h	Change RocksDB License	8 years ago
version_builder.cc	Revert "comment out unused parameters"	8 years ago
version_builder.h	Change RocksDB License	8 years ago
version_builder_test.cc	Change RocksDB License	8 years ago
version_edit.cc	Revert "comment out unused parameters"	8 years ago
version_edit.h	Change RocksDB License	8 years ago
version_edit_test.cc	Change RocksDB License	8 years ago
version_set.cc	Introduce bottom-pri thread pool for large universal compactions	7 years ago
version_set.h	Introduce bottom-pri thread pool for large universal compactions	7 years ago
version_set_test.cc	Revert "comment out unused parameters"	8 years ago
wal_manager.cc	Replace dynamic_cast<>	8 years ago
wal_manager.h	Change RocksDB License	8 years ago
wal_manager_test.cc	Revert "comment out unused parameters"	8 years ago
write_batch.cc	Revert "comment out unused parameters"	8 years ago
write_batch_base.cc	Change RocksDB License	8 years ago
write_batch_internal.h	Change RocksDB License	8 years ago
write_batch_test.cc	Gcc 7 ParsedInternalKey replace memset with clear function.	8 years ago
write_callback.h	Change RocksDB License	8 years ago
write_callback_test.cc	Fix flaky write_callback_test	8 years ago
write_controller.cc	Change RocksDB License	8 years ago
write_controller.h	Change RocksDB License	8 years ago
write_controller_test.cc	Change RocksDB License	8 years ago
write_thread.cc	Fix the overflow bug in AwaitState	7 years ago
write_thread.h	Change RocksDB License	8 years ago