fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Hui Xiao bab5f9a6f2 Add new stat rocksdb.table.open.prefetch.tail.read.bytes, rocksdb.table.open.prefetch.tail.{miss|hit} (#11265) ... Summary: **Context/Summary:** We are adding new stats to measure behavior of prefetched tail size and look up into this buffer The stat collection is done in FilePrefetchBuffer but only for prefetched tail buffer during table open for now using FilePrefetchBuffer enum. It's cleaner than the alternative of implementing in upper-level call places of FilePrefetchBuffer for table open. It also has the benefit of extensible to other types of FilePrefetchBuffer if needed. See db bench for perf regression concern. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11265 Test Plan: **- Piggyback on existing test** **- rocksdb.table.open.prefetch.tail.miss is harder to UT so I manually set prefetch tail read bytes to be small and run db bench.** ``` ./db_bench -db=/tmp/testdb -statistics=true -benchmarks="fillseq" -key_size=32 -value_size=512 -num=5000 -write_buffer_size=655 -target_file_size_base=655 -disable_auto_compactions=false -compression_type=none -bloom_bits=3 -use_direct_reads=true ``` ``` rocksdb.table.open.prefetch.tail.read.bytes P50 : 4096.000000 P95 : 4096.000000 P99 : 4096.000000 P100 : 4096.000000 COUNT : 225 SUM : 921600 rocksdb.table.open.prefetch.tail.miss COUNT : 91 rocksdb.table.open.prefetch.tail.hit COUNT : 1034 ``` **- No perf regression observed in db_bench** SETUP command: create same db with ~900 files for pre-change/post-change. ``` ./db_bench -db=/tmp/testdb -benchmarks="fillseq" -key_size=32 -value_size=512 -num=500000 -write_buffer_size=655360 -disable_auto_compactions=true -target_file_size_base=16777216 -compression_type=none ``` TEST command 60 runs or til convergence: as suggested by anand1976 and akankshamahajan15, vary `seek_nexts` and `async_io` in testing. ``` ./db_bench -use_existing_db=true -db=/tmp/testdb -statistics=false -cache_size=0 -cache_index_and_filter_blocks=false -benchmarks=seekrandom[-X60] -num=50000 -seek_nexts={10, 500, 1000} -async_io={0|1} -use_direct_reads=true ``` async io = 0, direct io read = true | seek_nexts = 10, 30 runs | seek_nexts = 500, 12 runs | seek_nexts = 1000, 6 runs -- | -- | -- | -- pre-post change | 4776 (± 28) ops/sec; 24.8 (± 0.1) MB/sec | 288 (± 1) ops/sec; 74.8 (± 0.4) MB/sec | 145 (± 4) ops/sec; 75.6 (± 2.2) MB/sec post-change | 4790 (± 32) ops/sec; 24.9 (± 0.2) MB/sec | 288 (± 3) ops/sec; 74.7 (± 0.8) MB/sec | 143 (± 3) ops/sec; 74.5 (± 1.6) MB/sec async io = 1, direct io read = true | seek_nexts = 10, 54 runs | seek_nexts = 500, 6 runs | seek_nexts = 1000, 4 runs -- | -- | -- | -- pre-post change | 3350 (± 36) ops/sec; 17.4 (± 0.2) MB/sec | 264 (± 0) ops/sec; 68.7 (± 0.2) MB/sec | 138 (± 1) ops/sec; 71.8 (± 1.0) MB/sec post-change | 3358 (± 27) ops/sec; 17.4 (± 0.1) MB/sec | 263 (± 2) ops/sec; 68.3 (± 0.8) MB/sec | 139 (± 1) ops/sec; 72.6 (± 0.6) MB/sec Reviewed By: ajkr Differential Revision: D43781467 Pulled By: hx235 fbshipit-source-id: a706a18472a8edb2b952bac3af40eec803537f2a		2 years ago
..
backup_engine_options.cc	Fix remaining uses of "backupable" (#9792)	3 years ago
backupenginejni.cc	Fix pointer to jlong conversion in 32 bit OS (#9396)	3 years ago
cache.cc	Put Cache and CacheWrapper in new public header (#11192)	2 years ago
cassandra_compactionfilterjni.cc	Fix pointer to jlong conversion in 32 bit OS (#9396)	3 years ago
cassandra_value_operator.cc	Fix pointer to jlong conversion in 32 bit OS (#9396)	3 years ago
checkpoint.cc	Fix pointer to jlong conversion in 32 bit OS (#9396)	3 years ago
clock_cache.cc	Revert NewClockCache signature (#10358)	2 years ago
columnfamilyhandle.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
compact_range_options.cc	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
compaction_filter.cc	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
compaction_filter_factory.cc	Fix pointer to jlong conversion in 32 bit OS (#9396)	3 years ago
compaction_filter_factory_jnicallback.cc	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
compaction_filter_factory_jnicallback.h	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
compaction_job_info.cc	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
compaction_job_stats.cc	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
compaction_options.cc	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
compaction_options_fifo.cc	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
compaction_options_universal.cc	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
comparator.cc	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
comparatorjnicallback.cc	Fix Java API ComparatorOptions use after delete error (#11176)	2 years ago
comparatorjnicallback.h	Fix Java API ComparatorOptions use after delete error (#11176)	2 years ago
compression_options.cc	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
concurrent_task_limiter.cc	Add some missing headers (#10519)	2 years ago
config_options.cc	Remove deprecated util functions in options_util.h (#11126)	2 years ago
cplusplus_to_java_convert.h	Fix pointer to jlong conversion in 32 bit OS (#9396)	3 years ago
env.cc	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
env_options.cc	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
event_listener.cc	Fix pointer to jlong conversion in 32 bit OS (#9396)	3 years ago
event_listener_jnicallback.cc	Add (& fix) some simple source code checks (#8821)	3 years ago
event_listener_jnicallback.h	Add (& fix) some simple source code checks (#8821)	3 years ago
filter.cc	Fix pointer to jlong conversion in 32 bit OS (#9396)	3 years ago
ingest_external_file_options.cc	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
iterator.cc	Extend Java RocksDB iterators to support indirect Byte Buffers (#9222)	3 years ago
jnicallback.cc	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
jnicallback.h	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
loggerjnicallback.cc	Fix pointer to jlong conversion in 32 bit OS (#9396)	3 years ago
loggerjnicallback.h	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
lru_cache.cc	Use the default metadata charge policy when creating an LRU cache via the Java API (#10577)	2 years ago
memory_util.cc	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
memtablejni.cc	Fix pointer to jlong conversion in 32 bit OS (#9396)	3 years ago
merge_operator.cc	Fix pointer to jlong conversion in 32 bit OS (#9396)	3 years ago
native_comparator_wrapper_test.cc	Fix pointer to jlong conversion in 32 bit OS (#9396)	3 years ago
optimistic_transaction_db.cc	RocksJava API - fix Transaction.multiGet() size limit, remove bogus EnsureLocalCapacity() calls (#10674)	2 years ago
optimistic_transaction_options.cc	Fix pointer to jlong conversion in 32 bit OS (#9396)	3 years ago
options.cc	Remove a couple deprecated convenience.h APIs (#11120)	2 years ago
options_util.cc	Remove deprecated util functions in options_util.h (#11126)	2 years ago
persistent_cache.cc	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
portal.h	Add new stat rocksdb.table.open.prefetch.tail.read.bytes, rocksdb.table.open.prefetch.tail.{miss|hit} (#11265)	2 years ago
ratelimiterjni.cc	Fix pointer to jlong conversion in 32 bit OS (#9396)	3 years ago
remove_emptyvalue_compactionfilterjni.cc	Fix pointer to jlong conversion in 32 bit OS (#9396)	3 years ago
restorejni.cc	Fix pointer to jlong conversion in 32 bit OS (#9396)	3 years ago
rocks_callback_object.cc	Fix tabs and lint-ignores (#6734)	5 years ago
rocksdb_exception_test.cc	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
rocksjni.cc	Improve Java API get() performance by reducing copies (#10970)	2 years ago
slice.cc	Fix pointer to jlong conversion in 32 bit OS (#9396)	3 years ago
snapshot.cc	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	5 years ago
sst_file_manager.cc	Fix pointer to jlong conversion in 32 bit OS (#9396)	3 years ago
sst_file_reader_iterator.cc	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
sst_file_readerjni.cc	Fix pointer to jlong conversion in 32 bit OS (#9396)	3 years ago
sst_file_writerjni.cc	Fix pointer to jlong conversion in 32 bit OS (#9396)	3 years ago
sst_partitioner.cc	Fix pointer to jlong conversion in 32 bit OS (#9396)	3 years ago
statistics.cc	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
statisticsjni.cc	Fix tabs and lint-ignores (#6734)	5 years ago
statisticsjni.h	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
table.cc	Remove compressed block cache (#11117)	2 years ago
table_filter.cc	Fix pointer to jlong conversion in 32 bit OS (#9396)	3 years ago
table_filter_jnicallback.cc	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
table_filter_jnicallback.h	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
testable_event_listener.cc	Update build files for java8 build (#9541)	3 years ago
thread_status.cc	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
trace_writer.cc	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
trace_writer_jnicallback.cc	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
trace_writer_jnicallback.h	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
transaction.cc	RocksJava API - fix Transaction.multiGet() size limit, remove bogus EnsureLocalCapacity() calls (#10674)	2 years ago
transaction_db.cc	RocksJava API - fix Transaction.multiGet() size limit, remove bogus EnsureLocalCapacity() calls (#10674)	2 years ago
transaction_db_options.cc	Fix pointer to jlong conversion in 32 bit OS (#9396)	3 years ago
transaction_log.cc	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
transaction_notifier.cc	Fix pointer to jlong conversion in 32 bit OS (#9396)	3 years ago
transaction_notifier_jnicallback.cc	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
transaction_notifier_jnicallback.h	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
transaction_options.cc	Fix pointer to jlong conversion in 32 bit OS (#9396)	3 years ago
ttl.cc	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
wal_filter.cc	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
wal_filter_jnicallback.cc	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
wal_filter_jnicallback.h	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
write_batch.cc	Fix pointer to jlong conversion in 32 bit OS (#9396)	3 years ago
write_batch_test.cc	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
write_batch_with_index.cc	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
write_buffer_manager.cc	Put Cache and CacheWrapper in new public header (#11192)	2 years ago
writebatchhandlerjnicallback.cc	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago
writebatchhandlerjnicallback.h	Run format check for *.h and *.cc files under java/ (#10851)	2 years ago