Summary:
Reads via mmap on concurrent workloads are much slower than pread.
For example on a 24-core server with storage that can do 100k IOPS or more
I can get no more than 10k IOPS with mmap reads and 32+ threads.
Test Plan: db_bench benchmarks
Reviewers: dhruba, heyongqiang
Reviewed By: heyongqiang
Differential Revision: https://reviews.facebook.net/D5433
Summary:
FLAGS_cache_size is a long, no need to scan %lld into a size_t
for it (which generates a compiler warning)
Test Plan: run db_bench
Reviewers: dhruba, heyongqiang
Reviewed By: heyongqiang
CC: heyongqiang
Differential Revision: https://reviews.facebook.net/D5427
Summary:
This adds an option to db_bench to specify the compression algorithm to
use for LevelDB
Test Plan: ran db_bench
Reviewers: dhruba
Reviewed By: dhruba
Differential Revision: https://reviews.facebook.net/D5421
Summary:
Ability to switch off filesystem read-aheads. This change is
backward-compatible: the default setting is to allow file
system read-aheads.
Test Plan: run benchmarks
Reviewers: heyongqiang, adsharma
Reviewed By: heyongqiang
Differential Revision: https://reviews.facebook.net/D5391
Summary:
When posix_fadvise(offset, offset) is usedm it frees up only those
pages in that specified range. But the filesystem could have done some
read-aheads and those get cached in the OS cache.
Do not cache readahead-pages in the OS cache.
Test Plan: run db_bench benchmark.
Reviewers: vamsi, heyongqiang
Reviewed By: heyongqiang
Differential Revision: https://reviews.facebook.net/D5379
Summary: Enable db_bench to specify block size.
Test Plan: compile and run
Reviewers: heyongqiang
Reviewed By: heyongqiang
Differential Revision: https://reviews.facebook.net/D5373
Summary: added a new option db_log_dir, which points the log dir. Inside that dir, in order to make log names unique, the log file name is prefixed with the leveldb data dir absolute path.
Test Plan: db_test
Reviewers: dhruba
Reviewed By: dhruba
Differential Revision: https://reviews.facebook.net/D5205
Summary: If none of reads or writes are specified by user, then pick the FLAGS_NUM as the number of iterations in the ReadRandomWriteRandom test. If either reads or writes are defined, then use their maximum.
Test Plan: run benchmark
Reviewers: heyongqiang
Reviewed By: heyongqiang
Differential Revision: https://reviews.facebook.net/D5217
Summary: Do not use scribe for release builds.
Test Plan: build fbcode
Reviewers: heyongqiang
Reviewed By: heyongqiang
Differential Revision: https://reviews.facebook.net/D5139
Summary:
This patch enables the db_bench benchmark to issue both random reads and random writes at the same time. This options can be trigged via
./db_bench --benchmarks=readrandomwriterandom
The default percetage of reads is 90.
One can change the percentage of reads by specifying the --readwritepercent.
./db_bench --benchmarks=readrandomwriterandom=50
This is a feature request from Jeffro asking for leveldb performance with a 90:10 read:write ratio.
Test Plan: run on test machine.
Reviewers: heyongqiang
Reviewed By: heyongqiang
Differential Revision: https://reviews.facebook.net/D5067
Summary:
Clean up compiler warnings generated by -Wall option.
make clean all OPT=-Wall
This is a pre-requisite before making a new release.
Test Plan: compile and run unit tests
Reviewers: heyongqiang
Reviewed By: heyongqiang
Differential Revision: https://reviews.facebook.net/D5019
Summary:
The numbers of shards that the block cache is divided into is
configurable. However, if the user specifies that he/she wants
the block cache to be divided into more than 2**20 pieces, then
the system will rey to allocate a huge array of that size) that
could fail.
It is better to limit the sharding of the block cache to an
upper bound. The default sharding is 16 shards (i.e. 2**4)
and the maximum is now 2 million shards (i.e. 2**20).
Also, fixed a bug with the LRUCache where the numShardBits
should be a private member of the LRUCache object rather than
a static variable.
Test Plan:
run db_bench with --cache_numshardbits=64.
Task ID: #
Blame Rev:
Reviewers: heyongqiang
Reviewed By: heyongqiang
Differential Revision: https://reviews.facebook.net/D5013
Summary: as subject. ported the change from google code leveldb 1.5
Test Plan: run db_test
Reviewers: dhruba
Differential Revision: https://reviews.facebook.net/D4839
Summary:
Introduce a new method Env->Fsync() that issues fsync (instead of fdatasync).
This is needed for data durability when running on ext3 filesystems.
Added options to the benchmark db_bench to generate performance numbers
with either fsync or fdatasync enabled.
Cleaned up Makefile to build leveldb_shell only when building the thrift
leveldb server.
Test Plan: build and run benchmark
Reviewers: heyongqiang
Reviewed By: heyongqiang
Differential Revision: https://reviews.facebook.net/D4911
Summary:
as subject
add a tool to read sst file
as subject.
./sst_reader --command=check --file=
./sst_reader --command=scan --file=
Test Plan:
db_test
run this command
Reviewers: dhruba
Reviewed By: dhruba
Differential Revision: https://reviews.facebook.net/D4881
Summary:
as subject
add a tool to read sst file
as subject.
./sst_reader --command=check --file=
./sst_reader --command=scan --file=
Test Plan:
db_test
run this command
Reviewers: dhruba
Differential Revision: https://reviews.facebook.net/D4881
Summary: Record the version of the source that we are compiling. We keep a record of the git revision in util/version.cc. This source file is then built as a regular source file as part of the compilation process. One can run "strings executable_filename | grep _build_" to find the version of the source that we used to build the executable file.
Test Plan: none
Differential Revision: https://reviews.facebook.net/D4785
Summary:
The fcntl call cannot detect lock conflicts when invoked multiple times
from the same thread.
Use a static lockedFile Set to record the paths that are locked.
A lockfile request checks to see if htis filename already exists in
lockedFiles, if so, then it triggers an error. Otherwise, it inserts
the filename in the lockedFiles Set.
A unlock file request verifies that the filename is in the lockedFiles
set and removes it from lockedFiles set.
Test Plan: unit test attached
Reviewers: heyongqiang
Reviewed By: heyongqiang
Differential Revision: https://reviews.facebook.net/D4755
Summary: implemented a commond line shell to talk with leveldb thrift server, which is based on state pattern and can be easily extended.
Test Plan: build and run
Reviewers: dhruba, zshao, heyongqiang
Differential Revision: https://reviews.facebook.net/D4713
Summary: Log the open-options to the LOG. Use options_ instead of options because SanitizeOptions could modify the max_file_open limit.
Test Plan: num db_bench
Reviewers: heyongqiang
Reviewed By: heyongqiang
Differential Revision: https://reviews.facebook.net/D4833
Summary:
Fixed unit test c_test by initializing logger=NULL.
Removed "atomic" from last_log_ts so that unit tests do not require C11 compiler.
Anyway, last_log_ts is mostly used for logging, so it is ok if it is loosely
accurate.
Test Plan: run c_test
Reviewers: heyongqiang
Reviewed By: heyongqiang
Differential Revision: https://reviews.facebook.net/D4803
Summary: Unit test corruption_test do not compile.
Test Plan: run unit tests
Reviewers: heyongqiang
Reviewed By: heyongqiang
Differential Revision: https://reviews.facebook.net/D4797
Summary: Record the version of the source that we are compiling. We keep a record of the git revision in util/version.cc. This source file is then built as a regular source file as part of the compilation process. One can run "strings executable_filename | grep _build_" to find the version of the source that we used to build the executable file.
Test Plan: none
Differential Revision: https://reviews.facebook.net/D4785
Summary:
as subject.
A new log is written to scribe via thrift client when a new db is opened and when there is
a compaction.
a new option var scribe_log_db_stats is added.
Test Plan: manually checked using command "ptail -time 0 leveldb_deploy_stats"
Reviewers: dhruba
Differential Revision: https://reviews.facebook.net/D4659
Summary:
The fcntl call cannot detect lock conflicts when invoked multiple times
from the same thread.
Use a static lockedFile Set to record the paths that are locked.
A lockfile request checks to see if htis filename already exists in
lockedFiles, if so, then it triggers an error. Otherwise, it inserts
the filename in the lockedFiles Set.
A unlock file request verifies that the filename is in the lockedFiles
set and removes it from lockedFiles set.
Test Plan: unit test attached
Reviewers: heyongqiang
Reviewed By: heyongqiang
Differential Revision: https://reviews.facebook.net/D4755
Summary: as subject and only maintain 10 log files.
Test Plan: new test in db_test
Reviewers: dhruba
Differential Revision: https://reviews.facebook.net/D4731
Summary: Expose new configration variables via the thrift api. when compiing for fbcode, always build thrift server
Test Plan: none.
Reviewers: heyongqiang
Reviewed By: heyongqiang
Differential Revision: https://reviews.facebook.net/D4689
Summary: The offsets used to speficy hostname and dbdir from the command line were not handled correctly.
Test Plan: none.
Reviewers: heyongqiang
Reviewed By: heyongqiang
Differential Revision: https://reviews.facebook.net/D4683
Mark Callaghan