fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Peter Dillinger 546e213c4f Fix DelayWrite() calls for two_write_queues (#11130) ... Summary: PR https://github.com/facebook/rocksdb/issues/11020 fixed a case where it was easy to deadlock the DB with LockWAL() but introduced a bug showing up as a rare assertion failure in the stress test. Specifically, `assert(w->state == STATE_INIT)` in `WriteThread::LinkOne()` called from `BeginWriteStall()`, `DelayWrite()`, `WriteImplWALOnly()`. I haven't been about to generate a unit test that reproduces this failure but I believe the root cause is that DelayWrite() was never meant to be re-entrant, only called from the DB's write_thread_ leader. https://github.com/facebook/rocksdb/issues/11020 introduced a call to DelayWrite() from the nonmem_write_thread_ group leader. This fix is to make DelayWrite() apply to the specific write queue that it is being called from (inject a dummy write stall entry to the head of the appropriate write queue). WriteController is re-entrant, based on polling and state changes signalled with bg_cv_, so can manage stalling two queues. The only anticipated complication (called out by Andrew in previous PR) is that we don't want timed write delays being injected in parallel for the two queues, because that dimishes the intended throttling effect. Thus, we only allow timed delays for the primary write queue. HISTORY not updated because this is intended for the same release where the bug was introduced. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11130 Test Plan: Although I was not able to reproduce the assertion failure, I was able to reproduce a distinct flaw with what I believe is the same root cause: a kind of deadlock if both write queues need to wake up from stopped writes. Only one will be waiting on bg_cv_ (the other waiting in `LinkOne()` for the write queue to open up), so a single SignalAll() will only unblock one of the queues, with the other re-instating the stop until another signal on bg_cv_. A simple unit test is added for this case. Will also run crash_test_with_multiops_wc_txn for a while looking for issues. Reviewed By: ajkr Differential Revision: D42749330 Pulled By: pdillinger fbshipit-source-id: 4317dd899a93d57c26fd5af7143038f82d4d4d1b		2 years ago
..
blob	Revert #10802 Consider range tombstone in compaction output file cutting (#11089)	2 years ago
compaction	Add API to limit blast radius of merge operator failure (#11092)	2 years ago
db_impl	Fix DelayWrite() calls for two_write_queues (#11130)	2 years ago
wide	Support Merge for wide-column entities in the compaction logic (#10946)	2 years ago
arena_wrapped_db_iter.cc	Ran clang-format on db/ directory (#10910)	2 years ago
arena_wrapped_db_iter.h	Ran clang-format on db/ directory (#10910)	2 years ago
builder.cc	Fix asan failure caused by range tombstone start key use-after-free (#11106)	2 years ago
builder.h	Include estimated bytes deleted by range tombstones in compensated file size (#10734)	2 years ago
c.cc	Remove compressed block cache (#11117)	2 years ago
c_test.c	Add C API for ReadOptions::async_io (#11062)	2 years ago
column_family.cc	Fix data race on `ColumnFamilyData::flush_reason` by letting FlushRequest/Job owns flush_reason instead of CFD (#11111)	2 years ago
column_family.h	Fix data race on `ColumnFamilyData::flush_reason` by letting FlushRequest/Job owns flush_reason instead of CFD (#11111)	2 years ago
column_family_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
compact_files_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
comparator_db_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
convenience.cc	Ran clang-format on db/ directory (#10910)	2 years ago
corruption_test.cc	Migrate ErrorEnv from EnvWrapper to FileSystemWrapper (#11124)	2 years ago
cuckoo_table_db_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
db_basic_test.cc	Remove compressed block cache (#11117)	2 years ago
db_block_cache_test.cc	Remove compressed block cache (#11117)	2 years ago
db_bloom_filter_test.cc	Add missing range conflict check between file ingestion and RefitLevel() (#10988)	2 years ago
db_compaction_filter_test.cc	Skip swaths of range tombstone covered keys in merging iterator (2022 edition) (#10449)	2 years ago
db_compaction_test.cc	Consider TTL compaction file cutting earlier to prevent small output file (#11075)	2 years ago
db_dynamic_level_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
db_encryption_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
db_filesnapshot.cc	Ran clang-format on db/ directory (#10910)	2 years ago
db_flush_test.cc	Fix data race on `ColumnFamilyData::flush_reason` by letting FlushRequest/Job owns flush_reason instead of CFD (#11111)	2 years ago
db_info_dumper.cc	Ran clang-format on db/ directory (#10910)	2 years ago
db_info_dumper.h	Add a DB Session ID (#6959)	5 years ago
db_inplace_update_test.cc	Fix in-place updates for value types other than kTypeValue (#10254)	3 years ago
db_io_failure_test.cc	Enable a few unit tests to use custom Env objects (#9087)	3 years ago
db_iter.cc	Add API to limit blast radius of merge operator failure (#11092)	2 years ago
db_iter.h	Support Merge with wide-column entities in iterator (#10941)	2 years ago
db_iter_stress_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
db_iter_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
db_iterator_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
db_kv_checksum_test.cc	Print stack traces on frozen tests in CI (#10828)	2 years ago
db_log_iter_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
db_logical_block_size_cache_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
db_memtable_test.cc	Add support for wide-column point lookups (#10540)	2 years ago
db_merge_operand_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
db_merge_operator_test.cc	Add API to limit blast radius of merge operator failure (#11092)	2 years ago
db_options_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
db_properties_test.cc	Major Cache refactoring, CPU efficiency improvement (#10975)	2 years ago
db_range_del_test.cc	Avoid counting extra range tombstone compensated size in `AddRangeDels()` (#11091)	2 years ago
db_rate_limiter_test.cc	Add rate-limiting support to batched MultiGet() (#10159)	3 years ago
db_readonly_with_timestamp_test.cc	Add timestamp support to CompactedDBImpl (#10030)	3 years ago
db_secondary_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
db_sst_test.cc	Fix periodic_task unable to re-register the same task type (#10379)	2 years ago
db_statistics_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
db_table_properties_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
db_tailing_iter_test.cc	Remove compressed block cache (#11117)	2 years ago
db_test.cc	Sort L0 files by newly introduced epoch_num (#10922)	2 years ago
db_test2.cc	Remove compressed block cache (#11117)	2 years ago
db_test_util.cc	Remove compressed block cache (#11117)	2 years ago
db_test_util.h	Remove compressed block cache (#11117)	2 years ago
db_universal_compaction_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
db_wal_test.cc	Revise LockWAL/UnlockWAL implementation (#11020)	2 years ago
db_with_timestamp_basic_test.cc	Fix asan failure caused by range tombstone start key use-after-free (#11106)	2 years ago
db_with_timestamp_compaction_test.cc	Fix an assertion failure in `TimestampTablePropertiesCollector` for empty output (#11015)	2 years ago
db_with_timestamp_test_util.cc	Add timestamp support to DBImplReadOnly (#10004)	3 years ago
db_with_timestamp_test_util.h	Add timestamp support to DBImplReadOnly (#10004)	3 years ago
db_write_buffer_manager_test.cc	Prevent a case of WriteBufferManager flush thrashing (#6364)	2 years ago
db_write_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
dbformat.cc	Remove copying of range tombstones keys in iterator (#10878)	2 years ago
dbformat.h	Remove copying of range tombstones keys in iterator (#10878)	2 years ago
dbformat_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
deletefile_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
error_handler.cc	Ran clang-format on db/ directory (#10910)	2 years ago
error_handler.h	Ran clang-format on db/ directory (#10910)	2 years ago
error_handler_fs_test.cc	Deflake DBErrorHandlingFSTest.*WALWriteError (#10642)	2 years ago
event_helpers.cc	Ran clang-format on db/ directory (#10910)	2 years ago
event_helpers.h	Ran clang-format on db/ directory (#10910)	2 years ago
experimental.cc	Include estimated bytes deleted by range tombstones in compensated file size (#10734)	2 years ago
external_sst_file_basic_test.cc	Fix some unit test failure in ExternalSSTFileBasicTest (#11070)	2 years ago
external_sst_file_ingestion_job.cc	Add missing range conflict check between file ingestion and RefitLevel() (#10988)	2 years ago
external_sst_file_ingestion_job.h	Add missing range conflict check between file ingestion and RefitLevel() (#10988)	2 years ago
external_sst_file_test.cc	Add missing range conflict check between file ingestion and RefitLevel() (#10988)	2 years ago
fault_injection_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
file_indexer.cc	Ran clang-format on db/ directory (#10910)	2 years ago
file_indexer.h	Ran clang-format on db/ directory (#10910)	2 years ago
file_indexer_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
filename_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
flush_job.cc	Fix data race on `ColumnFamilyData::flush_reason` by letting FlushRequest/Job owns flush_reason instead of CFD (#11111)	2 years ago
flush_job.h	Fix data race on `ColumnFamilyData::flush_reason` by letting FlushRequest/Job owns flush_reason instead of CFD (#11111)	2 years ago
flush_job_test.cc	Fix data race on `ColumnFamilyData::flush_reason` by letting FlushRequest/Job owns flush_reason instead of CFD (#11111)	2 years ago
flush_scheduler.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
flush_scheduler.h	Include C++ standard library headers instead of C compatibility headers (#8068)	4 years ago
forward_iterator.cc	Ran clang-format on db/ directory (#10910)	2 years ago
forward_iterator.h	Ran clang-format on db/ directory (#10910)	2 years ago
forward_iterator_bench.cc	Ran clang-format on db/ directory (#10910)	2 years ago
history_trimming_iterator.h	Revert #10802 Consider range tombstone in compaction output file cutting (#11089)	2 years ago
import_column_family_job.cc	Include estimated bytes deleted by range tombstones in compensated file size (#10734)	2 years ago
import_column_family_job.h	Add some missing headers (#10519)	2 years ago
import_column_family_test.cc	Use VersionBuilder for CF import ordering/validation (#11028)	2 years ago
internal_stats.cc	Major Cache refactoring, CPU efficiency improvement (#10975)	2 years ago
internal_stats.h	Major Cache refactoring, CPU efficiency improvement (#10975)	2 years ago
job_context.h	Ran clang-format on db/ directory (#10910)	2 years ago
kv_checksum.h	Add memtable per key-value checksum (#10281)	2 years ago
listener_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
log_format.h	Add record to set WAL compression type if enabled (#9556)	3 years ago
log_reader.cc	Ran clang-format on db/ directory (#10910)	2 years ago
log_reader.h	Ran clang-format on db/ directory (#10910)	2 years ago
log_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
log_writer.cc	Add manual_wal_flush, FlushWAL() to stress/crash test (#10698)	2 years ago
log_writer.h	Ran clang-format on db/ directory (#10910)	2 years ago
logs_with_prep_tracker.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
logs_with_prep_tracker.h	Ran clang-format on db/ directory (#10910)	2 years ago
lookup_key.h	Ran clang-format on db/ directory (#10910)	2 years ago
malloc_stats.cc	Ran clang-format on db/ directory (#10910)	2 years ago
malloc_stats.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
manual_compaction_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
memtable.cc	Add API to limit blast radius of merge operator failure (#11092)	2 years ago
memtable.h	Ran clang-format on db/ directory (#10910)	2 years ago
memtable_list.cc	Fix flush picking non-consecutive memtables (#10921)	2 years ago
memtable_list.h	Fix memtable-only iterator regression (#10705)	2 years ago
memtable_list_test.cc	Fix flush picking non-consecutive memtables (#10921)	2 years ago
merge_context.h	Ran clang-format on db/ directory (#10910)	2 years ago
merge_helper.cc	Add API to limit blast radius of merge operator failure (#11092)	2 years ago
merge_helper.h	Add API to limit blast radius of merge operator failure (#11092)	2 years ago
merge_helper_test.cc	Basic Support for Merge with user-defined timestamp (#10819)	2 years ago
merge_operator.cc	Ran clang-format on db/ directory (#10910)	2 years ago
merge_test.cc	Make the Env class Customizable (#9293)	3 years ago
obsolete_files_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
options_file_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
output_validator.cc	Cleanup includes in dbformat.h (#8930)	3 years ago
output_validator.h	Cleanup includes in dbformat.h (#8930)	3 years ago
perf_context_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
periodic_task_scheduler.cc	Fix periodic_task unable to re-register the same task type (#10379)	2 years ago
periodic_task_scheduler.h	Fix periodic_task unable to re-register the same task type (#10379)	2 years ago
periodic_task_scheduler_test.cc	Add an unittest for Periodic compaction conflict with ongoing compaction (#10908)	2 years ago
pinned_iterators_manager.h	Avoid allocations/copies for large `GetMergeOperands()` results (#10458)	2 years ago
plain_table_db_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
post_memtable_callback.h	Snapshots with user-specified timestamps (#9879)	3 years ago
pre_release_callback.h	Fix and detect headers with missing dependencies (#8893)	3 years ago
prefix_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
range_del_aggregator.cc	Ran clang-format on db/ directory (#10910)	2 years ago
range_del_aggregator.h	User-defined timestamp support for `DeleteRange()` (#10661)	2 years ago
range_del_aggregator_bench.cc	Improve FragmentTombstones() speed by lazily initializing `seq_set_` (#10848)	2 years ago
range_del_aggregator_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
range_tombstone_fragmenter.cc	Improve FragmentTombstones() speed by lazily initializing `seq_set_` (#10848)	2 years ago
range_tombstone_fragmenter.h	Improve FragmentTombstones() speed by lazily initializing `seq_set_` (#10848)	2 years ago
range_tombstone_fragmenter_test.cc	snapshots of FragmentedRangeTombstoneList must in ascending order (#11046)	2 years ago
read_callback.h	Fix and detect headers with missing dependencies (#8893)	3 years ago
repair.cc	Include estimated bytes deleted by range tombstones in compensated file size (#10734)	2 years ago
repair_test.cc	Sort L0 files by newly introduced epoch_num (#10922)	2 years ago
seqno_time_test.cc	Add an unittest for Periodic compaction conflict with ongoing compaction (#10908)	2 years ago
seqno_to_time_mapping.cc	Add option `preserve_internal_time_seconds` to preserve the time info (#10747)	2 years ago
seqno_to_time_mapping.h	Add option `preserve_internal_time_seconds` to preserve the time info (#10747)	2 years ago
snapshot_checker.h	Use STATIC_AVOID_DESTRUCTION for static objects with non-trivial destructors (#9958)	3 years ago
snapshot_impl.cc	Ran clang-format on db/ directory (#10910)	2 years ago
snapshot_impl.h	Ran clang-format on db/ directory (#10910)	2 years ago
table_cache.cc	Major Cache refactoring, CPU efficiency improvement (#10975)	2 years ago
table_cache.h	Major Cache refactoring, CPU efficiency improvement (#10975)	2 years ago
table_cache_sync_and_async.h	Major Cache refactoring, CPU efficiency improvement (#10975)	2 years ago
table_properties_collector.cc	Ran clang-format on db/ directory (#10910)	2 years ago
table_properties_collector.h	Fix an assertion failure in `TimestampTablePropertiesCollector` for empty output (#11015)	2 years ago
table_properties_collector_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
transaction_log_impl.cc	Ran clang-format on db/ directory (#10910)	2 years ago
transaction_log_impl.h	Ran clang-format on db/ directory (#10910)	2 years ago
trim_history_scheduler.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
trim_history_scheduler.h	Ran clang-format on db/ directory (#10910)	2 years ago
version_builder.cc	Major Cache refactoring, CPU efficiency improvement (#10975)	2 years ago
version_builder.h	Sort L0 files by newly introduced epoch_num (#10922)	2 years ago
version_builder_test.cc	Include estimated bytes deleted by range tombstones in compensated file size (#10734)	2 years ago
version_edit.cc	Include estimated bytes deleted by range tombstones in compensated file size (#10734)	2 years ago
version_edit.h	Avoid counting extra range tombstone compensated size in `AddRangeDels()` (#11091)	2 years ago
version_edit_handler.cc	Sort L0 files by newly introduced epoch_num (#10922)	2 years ago
version_edit_handler.h	Sort L0 files by newly introduced epoch_num (#10922)	2 years ago
version_edit_test.cc	Include estimated bytes deleted by range tombstones in compensated file size (#10734)	2 years ago
version_set.cc	Add API to limit blast radius of merge operator failure (#11092)	2 years ago
version_set.h	Major Cache refactoring, CPU efficiency improvement (#10975)	2 years ago
version_set_sync_and_async.h	Major Cache refactoring, CPU efficiency improvement (#10975)	2 years ago
version_set_test.cc	Include estimated bytes deleted by range tombstones in compensated file size (#10734)	2 years ago
version_util.h	Allow manifest fix-up without requiring prior state (#10796)	2 years ago
wal_edit.cc	Do not hold mutex when write keys if not necessary (#7516)	2 years ago
wal_edit.h	Do not hold mutex when write keys if not necessary (#7516)	2 years ago
wal_edit_test.cc	Do not hold mutex when write keys if not necessary (#7516)	2 years ago
wal_manager.cc	Ran clang-format on db/ directory (#10910)	2 years ago
wal_manager.h	Ran clang-format on db/ directory (#10910)	2 years ago
wal_manager_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
write_batch.cc	Add API to limit blast radius of merge operator failure (#11092)	2 years ago
write_batch_base.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
write_batch_internal.h	Ran clang-format on db/ directory (#10910)	2 years ago
write_batch_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
write_callback.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
write_callback_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
write_controller.cc	Revamp WriteController (#8064)	4 years ago
write_controller.h	Ran clang-format on db/ directory (#10910)	2 years ago
write_controller_test.cc	Ran clang-format on db/ directory (#10910)	2 years ago
write_thread.cc	Revise LockWAL/UnlockWAL implementation (#11020)	2 years ago
write_thread.h	Fix race in ExitAsBatchGroupLeader with pipelined writes (#9944)	2 years ago