fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

anand76 afa2420c2b Introduce a new storage specific Env API (#5761) ... Summary: The current Env API encompasses both storage/file operations, as well as OS related operations. Most of the APIs return a Status, which does not have enough metadata about an error, such as whether its retry-able or not, scope (i.e fault domain) of the error etc., that may be required in order to properly handle a storage error. The file APIs also do not provide enough control over the IO SLA, such as timeout, prioritization, hinting about placement and redundancy etc. This PR separates out the file/storage APIs from Env into a new FileSystem class. The APIs are updated to return an IOStatus with metadata about the error, as well as to take an IOOptions structure as input in order to allow more control over the IO. The user can set both ```options.env``` and ```options.file_system``` to specify that RocksDB should use the former for OS related operations and the latter for storage operations. Internally, a ```CompositeEnvWrapper``` has been introduced that inherits from ```Env``` and redirects individual methods to either an ```Env``` implementation or the ```FileSystem``` as appropriate. When options are sanitized during ```DB::Open```, ```options.env``` is replaced with a newly allocated ```CompositeEnvWrapper``` instance if both env and file_system have been specified. This way, the rest of the RocksDB code can continue to function as before. This PR also ports PosixEnv to the new API by splitting it into two - PosixEnv and PosixFileSystem. PosixEnv is defined as a sub-class of CompositeEnvWrapper, and threading/time functions are overridden with Posix specific implementations in order to avoid an extra level of indirection. The ```CompositeEnvWrapper``` translates ```IOStatus``` return code to ```Status```, and sets the severity to ```kSoftError``` if the io_status is retryable. The error handling code in RocksDB can then recover the DB automatically. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5761 Differential Revision: D18868376 Pulled By: anand1976 fbshipit-source-id: 39efe18a162ea746fabac6360ff529baba48486f		5 years ago
..
compaction	Introduce a new storage specific Env API (#5761)	5 years ago
db_impl	Introduce a new storage specific Env API (#5761)	5 years ago
arena_wrapped_db_iter.cc	Apply formatter on recent 45 commits. (#5827)	5 years ago
arena_wrapped_db_iter.h	Refactor ArenaWrappedDBIter into separate files (#5801)	5 years ago
blob_index.h	Mark BlobIndex::DebugString const	5 years ago
builder.cc	Introduce a new storage specific Env API (#5761)	5 years ago
builder.h	Introduce a new storage specific Env API (#5761)	5 years ago
c.cc	Apply formatter to some recent commits (#6138)	5 years ago
c_test.c	Fix c_test:filter for various CACHE_LINE_SIZEs (#6153)	5 years ago
column_family.cc	Introduce a new storage specific Env API (#5761)	5 years ago
column_family.h	Introduce a new storage specific Env API (#5761)	5 years ago
column_family_test.cc	Refactor trimming logic for immutable memtables (#5022)	5 years ago
compact_files_test.cc	Use aggregate initialization for FlushJobInfo/CompactionJobInfo (#5997)	5 years ago
compacted_db_impl.cc	New API to get all merge operands for a Key (#5604)	5 years ago
compacted_db_impl.h	Use delete to disable automatic generated methods. (#5009)	5 years ago
comparator_db_test.cc	Move some memory related files from util/ to memory/ (#5382)	6 years ago
convenience.cc	Introduce a new storage specific Env API (#5761)	5 years ago
corruption_test.cc	Introduce a new storage specific Env API (#5761)	5 years ago
cuckoo_table_db_test.cc	Fix memory leak on error opening PlainTable (#5951)	5 years ago
db_basic_test.cc	Apply formatter to some recent commits (#6138)	5 years ago
db_blob_index_test.cc	Disable blob iterator test with max_sequential_skip_in_iterations==0 in LITE mode (#6052)	5 years ago
db_block_cache_test.cc	Charge block cache for cache internal usage (#5797)	5 years ago
db_bloom_filter_test.cc	Expose and elaborate FilterBuildingContext (#6088)	5 years ago
db_compaction_filter_test.cc	upgrade gtest 1.7.0 => 1.8.1 for json result writing	5 years ago
db_compaction_test.cc	wait pending memtable writes on file ingestion or compact range (#6113)	5 years ago
db_dynamic_level_test.cc	Fix flaky DBDynamicLevelTest.DynamicLevelMaxBytesBase2 (#4668)	6 years ago
db_encryption_test.cc	Fix EncryptedEnv assert (#5735)	5 years ago
db_filesnapshot.cc	Support concurrent CF iteration and drop (#6147)	5 years ago
db_flush_test.cc	Fix a potential bug scheduling unnecessary threads (#6104)	5 years ago
db_info_dumper.cc	Apply formatter to recent 200+ commits. (#5830)	5 years ago
db_info_dumper.h	Change RocksDB License	7 years ago
db_inplace_update_test.cc	Change RocksDB License	7 years ago
db_io_failure_test.cc	Disable DBIOFailureTest.NoSpaceCompactRange in LITE (#4596)	6 years ago
db_iter.cc	Fix blob context when db_iter uses seek (#6051)	5 years ago
db_iter.h	Apply formatter on recent 45 commits. (#5827)	5 years ago
db_iter_stress_test.cc	Move some memory related files from util/ to memory/ (#5382)	6 years ago
db_iter_test.cc	Move some memory related files from util/ to memory/ (#5382)	6 years ago
db_iterator_test.cc	Add useful idioms to Random API (OneInOpt, PercentTrue) (#6154)	5 years ago
db_log_iter_test.cc	Apply modernize-use-override (2nd iteration)	6 years ago
db_memtable_test.cc	upgrade gtest 1.7.0 => 1.8.1 for json result writing	5 years ago
db_merge_operand_test.cc	New API to get all merge operands for a Key (#5604)	5 years ago
db_merge_operator_test.cc	New API to get all merge operands for a Key (#5604)	5 years ago
db_options_test.cc	Make default value of options.ttl to be 30 days when it is supported. (#6073)	5 years ago
db_properties_test.cc	Refactor/consolidate legacy Bloom implementation details (#5784)	5 years ago
db_range_del_test.cc	Fix RangeDeletion bug (#6062)	5 years ago
db_sst_test.cc	Fix bugs in WAL trash file handling (#5520)	5 years ago
db_statistics_test.cc	Make statistics's stats_level change thread-safe (#5030)	6 years ago
db_table_properties_test.cc	upgrade gtest 1.7.0 => 1.8.1 for json result writing	5 years ago
db_tailing_iter_test.cc	Remove managed iterator	6 years ago
db_test.cc	Support options.max_open_files = -1 with periodic_compaction_seconds (#6090)	5 years ago
db_test2.cc	Fix a regression bug on total order seek with prefix enabled and range delete (#6028)	5 years ago
db_test_util.cc	Batched MultiGet API for multiple column families (#5816)	5 years ago
db_test_util.h	Expose and elaborate FilterBuildingContext (#6088)	5 years ago
db_universal_compaction_test.cc	Use SpecialSkipListFactory in RecalculateScoreAfterPicking (#6125)	5 years ago
db_wal_test.cc	Introduce a new storage specific Env API (#5761)	5 years ago
db_write_test.cc	Move some memory related files from util/ to memory/ (#5382)	6 years ago
dbformat.cc	Apply formatter to recent 200+ commits. (#5830)	5 years ago
dbformat.h	Use delete to disable automatic generated methods. (#5009)	5 years ago
dbformat_test.cc	Move some logging related files to logging/ (#5387)	6 years ago
deletefile_test.cc	Refactor deletefile_test.cc (#5822)	5 years ago
error_handler.cc	Make format	6 years ago
error_handler.h	Fix typos in comments (#4456)	6 years ago
error_handler_test.cc	wait pending memtable writes on file ingestion or compact range (#6113)	5 years ago
event_helpers.cc	BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903)	5 years ago
event_helpers.h	BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903)	5 years ago
experimental.cc	Organizing rocksdb/db directory	6 years ago
external_sst_file_basic_test.cc	Allow ingesting overlapping files (#5539)	5 years ago
external_sst_file_ingestion_job.cc	Introduce a new storage specific Env API (#5761)	5 years ago
external_sst_file_ingestion_job.h	Introduce a new storage specific Env API (#5761)	5 years ago
external_sst_file_test.cc	wait pending memtable writes on file ingestion or compact range (#6113)	5 years ago
fault_injection_test.cc	Move some logging related files to logging/ (#5387)	6 years ago
file_indexer.cc	Change RocksDB License	7 years ago
file_indexer.h	Move some memory related files from util/ to memory/ (#5382)	6 years ago
file_indexer_test.cc	Move some memory related files from util/ to memory/ (#5382)	6 years ago
filename_test.cc	Move some logging related files to logging/ (#5387)	6 years ago
flush_job.cc	Introduce a new storage specific Env API (#5761)	5 years ago
flush_job.h	Introduce a new storage specific Env API (#5761)	5 years ago
flush_job_test.cc	Introduce a new storage specific Env API (#5761)	5 years ago
flush_scheduler.cc	Support concurrent CF iteration and drop (#6147)	5 years ago
flush_scheduler.h	Refactor trimming logic for immutable memtables (#5022)	5 years ago
forward_iterator.cc	Add more callers for table reader. (#5454)	5 years ago
forward_iterator.h	Move some memory related files from util/ to memory/ (#5382)	6 years ago
forward_iterator_bench.cc	simplify include directive involving inttypes (#5402)	6 years ago
import_column_family_job.cc	Introduce a new storage specific Env API (#5761)	5 years ago
import_column_family_job.h	Introduce a new storage specific Env API (#5761)	5 years ago
import_column_family_test.cc	Apply formatter to recent 200+ commits. (#5830)	5 years ago
internal_stats.cc	Apply formatter to recent 200+ commits. (#5830)	5 years ago
internal_stats.h	Rename InternalDBStatsType enum names (#5779)	5 years ago
job_context.h	WritePrepared: Fix visible key compacted out by compaction (#4883)	6 years ago
listener_test.cc	Propagate SST and blob file numbers through the EventListener interface (#5962)	5 years ago
log_format.h	Fix an inaccurate comment (#4315)	6 years ago
log_reader.cc	Divide file_reader_writer.h and .cc (#5803)	5 years ago
log_reader.h	Divide file_reader_writer.h and .cc (#5803)	5 years ago
log_test.cc	Introduce a new storage specific Env API (#5761)	5 years ago
log_writer.cc	Divide file_reader_writer.h and .cc (#5803)	5 years ago
log_writer.h	Use delete to disable automatic generated methods. (#5009)	5 years ago
logs_with_prep_tracker.cc	Skip deleted WALs during recovery	7 years ago
logs_with_prep_tracker.h	Skip deleted WALs during recovery	7 years ago
lookup_key.h	Avoid user key copying for Get/Put/Write with user-timestamp (#5502)	5 years ago
malloc_stats.cc	Support jemalloc compiled with `--with-jemalloc-prefix` (#5521)	5 years ago
malloc_stats.h	Change RocksDB License	7 years ago
manual_compaction_test.cc	Move some memory related files from util/ to memory/ (#5382)	6 years ago
memtable.cc	Misc hashing updates / upgrades (#5909)	5 years ago
memtable.h	Fix OnFlushCompleted fired before flush result write to MANIFEST (#5908)	5 years ago
memtable_list.cc	Fix RangeDeletion bug (#6062)	5 years ago
memtable_list.h	Fix OnFlushCompleted fired before flush result write to MANIFEST (#5908)	5 years ago
memtable_list_test.cc	Introduce a new storage specific Env API (#5761)	5 years ago
merge_context.h	Introduce a new MultiGet batching implementation (#5011)	6 years ago
merge_helper.cc	Fix merging range tombstone covering put during flush/compaction (#5406)	6 years ago
merge_helper.h	Remove v1 RangeDelAggregator (#4778)	6 years ago
merge_helper_test.cc	Move some memory related files from util/ to memory/ (#5382)	6 years ago
merge_operator.cc	Change RocksDB License	7 years ago
merge_test.cc	Make format	6 years ago
obsolete_files_test.cc	Refactor ObsoleteFilesTest to inherit from DBTestBase (#5820)	5 years ago
options_file_test.cc	Organizing rocksdb/db directory	6 years ago
perf_context_test.cc	Move some memory related files from util/ to memory/ (#5382)	6 years ago
pinned_iterators_manager.h	Change RocksDB License	7 years ago
plain_table_db_test.cc	Fix memory leak on error opening PlainTable (#5951)	5 years ago
pre_release_callback.h	WritePrepared: reduce prepared_mutex_ overhead (#5420)	6 years ago
prefix_test.cc	Organizing rocksdb/db directory	6 years ago
range_del_aggregator.cc	Organizing rocksdb/db directory	6 years ago
range_del_aggregator.h	Organizing rocksdb/db directory	6 years ago
range_del_aggregator_bench.cc	Move some memory related files from util/ to memory/ (#5382)	6 years ago
range_del_aggregator_test.cc	Move test related files under util/ to test_util/ (#5377)	6 years ago
range_tombstone_fragmenter.cc	Apply formatter to recent 200+ commits. (#5830)	5 years ago
range_tombstone_fragmenter.h	Initialized pinned_pos_ and pinned_seq_pos_ in FragmentedRangeTombstoneIterator (#5720)	5 years ago
range_tombstone_fragmenter_test.cc	Move test related files under util/ to test_util/ (#5377)	6 years ago
read_callback.h	WriteUnPrepared: improve read your own write functionality (#5573)	5 years ago
repair.cc	Introduce a new storage specific Env API (#5761)	5 years ago
repair_test.cc	Introduce a new storage specific Env API (#5761)	5 years ago
snapshot_checker.h	WritePrepared: fix issue with snapshot released during compaction (#4858)	6 years ago
snapshot_impl.cc	Change RocksDB License	7 years ago
snapshot_impl.h	Refresh snapshot list during long compactions (2nd attempt) (#5278)	6 years ago
table_cache.cc	Introduce a new storage specific Env API (#5761)	5 years ago
table_cache.h	Introduce a new storage specific Env API (#5761)	5 years ago
table_properties_collector.cc	Feature for sampling and reporting compressibility (#4842)	6 years ago
table_properties_collector.h	Feature for sampling and reporting compressibility (#4842)	6 years ago
table_properties_collector_test.cc	Introduce a new storage specific Env API (#5761)	5 years ago
transaction_log_impl.cc	Introduce a new storage specific Env API (#5761)	5 years ago
transaction_log_impl.h	reuse scratch buffer in transaction_log_reader (#5702)	5 years ago
trim_history_scheduler.cc	Support concurrent CF iteration and drop (#6147)	5 years ago
trim_history_scheduler.h	Refactor trimming logic for immutable memtables (#5022)	5 years ago
version_builder.cc	Introduce a new storage specific Env API (#5761)	5 years ago
version_builder.h	Introduce a new storage specific Env API (#5761)	5 years ago
version_builder_test.cc	Support options.max_open_files = -1 with periodic_compaction_seconds (#6090)	5 years ago
version_edit.cc	Support options.max_open_files = -1 with periodic_compaction_seconds (#6090)	5 years ago
version_edit.h	Support options.max_open_files = -1 with periodic_compaction_seconds (#6090)	5 years ago
version_edit_test.cc	Support options.max_open_files = -1 with periodic_compaction_seconds (#6090)	5 years ago
version_set.cc	Introduce a new storage specific Env API (#5761)	5 years ago
version_set.h	Introduce a new storage specific Env API (#5761)	5 years ago
version_set_test.cc	Introduce a new storage specific Env API (#5761)	5 years ago
wal_manager.cc	Introduce a new storage specific Env API (#5761)	5 years ago
wal_manager.h	Introduce a new storage specific Env API (#5761)	5 years ago
wal_manager_test.cc	Introduce a new storage specific Env API (#5761)	5 years ago
write_batch.cc	Apply formatter to recent 200+ commits. (#5830)	5 years ago
write_batch_base.cc	Change RocksDB License	7 years ago
write_batch_internal.h	Add insert hints for each writebatch (#5728)	5 years ago
write_batch_test.cc	upgrade gtest 1.7.0 => 1.8.1 for json result writing	5 years ago
write_callback.h	Change RocksDB License	7 years ago
write_callback_test.cc	WritePrepared: reduce prepared_mutex_ overhead (#5420)	6 years ago
write_controller.cc	Change RocksDB License	7 years ago
write_controller.h	Change RocksDB License	7 years ago
write_controller_test.cc	Move test related files under util/ to test_util/ (#5377)	6 years ago
write_thread.cc	Option to make write group size configurable (#5759)	5 years ago
write_thread.h	Option to make write group size configurable (#5759)	5 years ago