fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Mike Kolupaev 637e64b9ac Add an option to prevent DB::Open() from querying sizes of all sst files (#6353) ... Summary: When paranoid_checks is on, DBImpl::CheckConsistency() iterates over all sst files and calls Env::GetFileSize() for each of them. As far as I could understand, this is pretty arbitrary and doesn't affect correctness - if filesystem doesn't corrupt fsynced files, the file sizes will always match; if it does, it may as well corrupt contents as well as sizes, and rocksdb doesn't check contents on open. If there are thousands of sst files, getting all their sizes takes a while. If, on top of that, Env is overridden to use some remote storage instead of local filesystem, it can be *really* slow and overload the remote storage service. This PR adds an option to not do GetFileSize(); instead it does GetChildren() for parent directory to check that all the expected sst files are at least present, but doesn't check their sizes. We can't just disable paranoid_checks instead because paranoid_checks do a few other important things: make the DB read-only on write errors, print error messages on read errors, etc. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6353 Test Plan: ran the added sanity check unit test. Will try it out in a LogDevice test cluster where the GetFileSize() calls are causing a lot of trouble. Differential Revision: D19656425 Pulled By: al13n321 fbshipit-source-id: c2c421b367633033760d1f56747bad206d1fbf82		5 years ago
..
backupablejni.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
backupenginejni.cc	Support range deletion tombstones in IngestExternalFile SSTs (#3778)	6 years ago
cassandra_compactionfilterjni.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
cassandra_value_operator.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
checkpoint.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
clock_cache.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
columnfamilyhandle.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
compact_range_options.cc	Add CompactRangeOptions for Java (#4220)	6 years ago
compaction_filter.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
compaction_filter_factory.cc	Add missing functionality to RocksJava (#4833)	6 years ago
compaction_filter_factory_jnicallback.cc	Added CompactionFilterFactory support to RocksJava	7 years ago
compaction_filter_factory_jnicallback.h	Added CompactionFilterFactory support to RocksJava	7 years ago
compaction_job_info.cc	Add missing functionality to RocksJava (#4833)	6 years ago
compaction_job_stats.cc	Add missing functionality to RocksJava (#4833)	6 years ago
compaction_options.cc	Add missing functionality to RocksJava (#4833)	6 years ago
compaction_options_fifo.cc	Add missing functionality to RocksJava (#4833)	6 years ago
compaction_options_universal.cc	Add missing functionality to RocksJava (#4833)	6 years ago
comparator.cc	Improve RocksJava Comparator (#6252)	5 years ago
comparatorjnicallback.cc	Improve RocksJava Comparator (#6252)	5 years ago
comparatorjnicallback.h	Improve RocksJava Comparator (#6252)	5 years ago
compression_options.cc	Add missing functionality to RocksJava (#4833)	6 years ago
env.cc	Add missing functionality to RocksJava (#4833)	6 years ago
env_options.cc	Add missing functionality to RocksJava (#4833)	6 years ago
filter.cc	Allow fractional bits/key in BloomFilterPolicy (#6092)	5 years ago
ingest_external_file_options.cc	Add missing functionality to RocksJava (#4833)	6 years ago
iterator.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
jnicallback.cc	Suppress lint in old files	7 years ago
jnicallback.h	Suppress lint in old files	7 years ago
loggerjnicallback.cc	Remove invalid comparison of va_list and nullptr (#5836)	5 years ago
loggerjnicallback.h	Added CompactionFilterFactory support to RocksJava	7 years ago
lru_cache.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
memory_util.cc	Add missing functionality to RocksJava (#4833)	6 years ago
memtablejni.cc	Add missing functionality to RocksJava (#4833)	6 years ago
merge_operator.cc	Add copyright headers per FB open-source checkup tool. (#5199)	6 years ago
native_comparator_wrapper_test.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
optimistic_transaction_db.cc	Add missing functionality to RocksJava (#4833)	6 years ago
optimistic_transaction_options.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
options.cc	Add an option to prevent DB::Open() from querying sizes of all sst files (#6353)	5 years ago
options_util.cc	Add missing functionality to RocksJava (#4833)	6 years ago
persistent_cache.cc	Add missing functionality to RocksJava (#4833)	6 years ago
portal.h	Improve RocksJava Comparator (#6252)	5 years ago
ratelimiterjni.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
remove_emptyvalue_compactionfilterjni.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
restorejni.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
rocks_callback_object.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
rocksdb_exception_test.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
rocksjni.cc	RocksDB#keyMayExist should not assume database values are unicode strings (#6186)	5 years ago
slice.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
snapshot.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
sst_file_manager.cc	Add missing functionality to RocksJava (#4833)	6 years ago
sst_file_reader_iterator.cc	Apply formatter on recent 45 commits. (#5827)	5 years ago
sst_file_readerjni.cc	Apply formatter on recent 45 commits. (#5827)	5 years ago
sst_file_writerjni.cc	Improve RocksJava Comparator (#6252)	5 years ago
statistics.cc	Make statistics's stats_level change thread-safe (#5030)	6 years ago
statisticsjni.cc	Get `CompactionJobInfo` from CompactFiles	6 years ago
statisticsjni.h	Suppress lint in old files	7 years ago
table.cc	JNI: Do not create 8M block cache for negative blockCacheSize values (#5465)	5 years ago
table_filter.cc	Add missing functionality to RocksJava (#4833)	6 years ago
table_filter_jnicallback.cc	Add missing functionality to RocksJava (#4833)	6 years ago
table_filter_jnicallback.h	Add missing functionality to RocksJava (#4833)	6 years ago
thread_status.cc	Add missing functionality to RocksJava (#4833)	6 years ago
trace_writer.cc	Add missing functionality to RocksJava (#4833)	6 years ago
trace_writer_jnicallback.cc	Add missing functionality to RocksJava (#4833)	6 years ago
trace_writer_jnicallback.h	Add missing functionality to RocksJava (#4833)	6 years ago
transaction.cc	Extend Transaction::GetForUpdate with do_validate (#4680)	6 years ago
transaction_db.cc	Add missing functionality to RocksJava (#4833)	6 years ago
transaction_db_options.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
transaction_log.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
transaction_notifier.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
transaction_notifier_jnicallback.cc	Add TransactionDB and OptimisticTransactionDB to the Java API	7 years ago
transaction_notifier_jnicallback.h	Add TransactionDB and OptimisticTransactionDB to the Java API	7 years ago
transaction_options.cc	comment unused parameters to turn on -Wunused-parameter flag	7 years ago
ttl.cc	Add missing functionality to RocksJava (#4833)	6 years ago
wal_filter.cc	Add missing functionality to RocksJava (#4833)	6 years ago
wal_filter_jnicallback.cc	Add missing functionality to RocksJava (#4833)	6 years ago
wal_filter_jnicallback.h	Add missing functionality to RocksJava (#4833)	6 years ago
write_batch.cc	Move some logging related files to logging/ (#5387)	6 years ago
write_batch_test.cc	upgrade gtest 1.7.0 => 1.8.1 for json result writing	5 years ago
write_batch_with_index.cc	Improve RocksJava Comparator (#6252)	5 years ago
write_buffer_manager.cc	Plumb WriteBufferManager through JNI (#4492)	6 years ago
writebatchhandlerjnicallback.cc	Remove warnings caused by unused variables in jni (#4345)	6 years ago
writebatchhandlerjnicallback.h	WriteUnPrepared: Add support for recovering WriteUnprepared transactions (#4078)	6 years ago