Summary:
When building rocksdbjava and rocksdbjavastatic, create -fPIC-enabled
binaries in a temporary subdirectory, jl/.
* Makefile (java_libobjects): New variable.
(java_libobjects): New rule.
(CLEAN_FILES): Arrange for "make clean" to remove that temporary dir.
(rocksdbjavastatic): Depend on the new variable.
Remove useless OPT=... line.
(rocksdbjava): Likewise.
Test Plan:
JAVA_HOME=/usr/local/jdk-7u67-64 PATH=$JAVA_HOME/bin:$PATH \
make rocksdbjavastatic
Reviewers: yhchiang
Reviewed By: yhchiang
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D36645
Summary:
* src.mk (JNI_NATIVE_SOURCES): New variable, so we don't have to use
a glob in Makefile
* Makefile (JNI_NATIVE_SOURCES): Remove glob-using definition, now
that the explicit list of sources is in src.mk.
Test Plan:
Run this:
JAVA_HOME=/usr/local/jdk-7u67-64 PATH=$JAVA_HOME/bin:$PATH \
make rocksdbjava
Reviewers: yhchiang
Reviewed By: yhchiang
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D36633
Summary:
As described in https://github.com/facebook/rocksdb/issues/563, we should add minor version to SONAME, since we break ABI with minor releases.
I also turned PLATFORM_SHARED_VERSIONED to true by default. This is true in LevelDB and it was switched to false by D15117 for no apparent reason. It should only be false for iOS.
Test Plan: `make shared_lib` produced librocksdb.dylib.3.10.0
Reviewers: sdong, yhchiang, meyering
Reviewed By: meyering
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D36573
Summary:
When GNU parallel is available, "make check" tests are now run in parallel.
When /dev/shm is usable, we tell those tests to create temporary files therein.
Now, the longest-running single test, db_test, (which is composed of hundreds of sub-tests)
is no longer run sequentially: instead, each of its sub-tests is run independently, and can
be parallelized along with all other tests. To make that process easier, this change
creates a temporary directory, "t/", in which it puts a small script for each of those
subtests. The output from each parallel-run test is now saved in t/log-TEST_NAME.
When GNU parallel is not available, we run the tests in sequence, just as before.
If GNU parallel is available and you don't like the default of running one subtest
per core, you can invoke "make J=1 check" to run only one test at a time.
Beware: this will take a long time, and it starts with the two longest-running tests, so you
will wait for a long time before seeing any results. Instead, if you want to use fewer resources
but still see useful progress, try "make J=60% check". That will attempt to ensure that 60% of
the cores are occupied by test runs.
To watch progress of individual tests (duration, success (PASS-or-FAIL), name), run "make watch-log"
in the same directory from another window. That will start with something like this:
and when complete should show numbers/names like this:
Every 0.1s: sort -k7,7nr -k4,4gr LOG|perl -n -e '@a=split("\t",$_,-1); $t=$a[8]; $t =~ s,^\./,,;' -e '$t =~ s, >.*,,; chomp $t;' -e '$t =~ /.*--gtest_filter=... Wed Apr 1 10:51:42 2015
152.221 PASS t/DBTest.FileCreationRandomFailure
109.280 PASS t/DBTest.EncodeDecompressedBlockSizeTest
82.315 PASS reduce_levels_test
77.812 PASS t/DBTest.CompactionFilterWithValueChange
73.236 PASS backupable_db_test
63.428 PASS deletefile_test
57.248 PASS table_test
55.665 PASS prefix_test
49.816 PASS t/DBTest.RateLimitingTest
...
Test Plan:
Timings (measured so as to exclude compile and link times):
With this change, all tests complete in 2m40s on a system for which nproc prints 32.
Prior to this this change, "make check" would take 24.5 minutes on that same system.
Here are durations (in seconds) of the longest-running subtests:
152.435 PASS t/DBTest.FileCreationRandomFailure
107.070 PASS t/DBTest.EncodeDecompressedBlockSizeTest
81.391 PASS ./reduce_levels_test
71.587 PASS ./backupable_db_test
61.746 PASS ./deletefile_test
57.960 PASS ./table_test
55.230 PASS ./prefix_test
54.060 PASS t/DBTest.CompactionFilterWithValueChange
48.873 PASS t/DBTest.RateLimitingTest
47.569 PASS ./fault_injection_test
46.593 PASS t/DBTest.Randomized
42.662 PASS t/DBTest.CompactionFilter
31.793 PASS t/DBTest.SparseMerge
30.612 PASS t/DBTest.CompactionFilterV2
25.891 PASS t/DBTest.GroupCommitTest
23.863 PASS t/DBTest.DynamicLevelMaxBytesBase
22.976 PASS ./rate_limiter_test
18.942 PASS t/DBTest.OptimizeFiltersForHits
16.851 PASS ./env_test
15.399 PASS t/DBTest.CompactionFilterV2WithValueChange
14.827 PASS t/DBTest.CompactionFilterV2NULLPrefix
Reviewers: igor, sdong, rven, yhchiang, igor.sugak
Reviewed By: igor.sugak
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D35379
Summary:
Just couple of small changes:
1. removed signal_test, since it doesn't seem useful and we don't even run it as part of `make check`
2. moved perf_context_test to TESTS instead of PROGRAMS
3. `make release` probably shouldn't compile benchmarks. We currently rely on `make release` building db_bench (via Jenkins), so I left db_bench there.
This is just a minor cleanup. We need to rethink our targets since they are a bit messy right now. We can do this during our tech debt week.
Test Plan: make release
Reviewers: anthony, rven, yhchiang, sdong, meyering
Reviewed By: meyering
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D36171
Summary:
This fixes two bugs: "make clean" would never remove the generated
file, util/build_version.cc, and since D33591, would be regenerated
only if it were absent.
* Makefile (clean): Remove the generated file.
(util/build_version.cc): Depend on the no-prereq FORCE target,
so that this target's rules are always run.
Since this is a generated file, make it read-only.
Also, be sure to remove the temporary file when it is the same
as the original.
Test Plan:
Ensure that we attempt regeneration every time.
Make it empty with an up-to-date time stamp and demonstrate
that it is rebuilt with the expected content:
$ : > util/build_version.cc
$ make util/build_version.o
GEN util/build_version.cc
GEN util/build_version.d
GEN util/build_version.cc
CC util/build_version.o
$ cat util/build_version.cc
#include "build_version.h"
const char* rocksdb_build_git_sha = "rocksdb_build_git_sha:v3.10-2-gb30e72a";
const char* rocksdb_build_git_date = "rocksdb_build_git_date:2015-03-27";
const char* rocksdb_build_compile_date = __DATE__;
Reviewers: igor.sugak, sdong, ljin, igor, rven
Reviewed By: rven
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D36087
Summary:
* Makefile (COMPILE_WITH_TSAN): Avoid a link failure by disabling
-pg when building with TSAN enabled.
Now that "make check" builds all $(PROGRAMS), it is linking
a few programs that were not normally linked before.
For example, this would fail to link with the following diagnostic:
COMPILE_WITH_TSAN=1 make -j40 log_and_apply_bench
CCLD log_and_apply_bench
ld: /usr/lib/../lib64/gcrt1.o: relocation R_X86_64_32S against `__libc_csu_fini' can not be used when making a shared object; recompile with -fPIC
/usr/lib/../lib64/gcrt1.o: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
Makefile:511: recipe for target 'log_and_apply_bench' failed
make: *** [log_and_apply_bench] Error 1
Since removing -pg is sufficient to get past this link
failure, and no one cares about profiling TSAN-enabled
binaries anyway, we will refrain from linking with -pg
when TSAN testing is enabled. Use a new variable, "pg"
which is set to "-pg" in most cases, but that is made
empty when COMPILE_WITH_TSAN is set.
Test Plan:
Now, this succeeds:
rm -f log_and_apply_bench
COMPILE_WITH_TSAN=1 make -j40 log_and_apply_bench
Reviewers: igor.sugak, rven, sdong, ljin, igor
Reviewed By: igor
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D35943
Summary:
* Makefile (check): Cause "make check" to build all $(PROGRAMS),
so that it verifies that the few benchmark-only source files that
not already built via "make check" do compile and link successfully.
Test Plan:
run "make clean; make check", and verify that
table/table_reader_bench.cc is now compiled.
Before, it was not, which led to an incomplete fix
for a build break.
Reviewers: ljin, igor.sugak, rven, igor, sdong
Reviewed By: sdong
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D35883
Summary:
changed make unity target to use $LIB_SOURCES as a source of library source code. In the old way (using find) table/mock_table.h was added to a list of library objects and this was a course of `make unity` break.
`build_tools/unity` contains some redundant code, I deleted it and moved the functionality in the Makefile.
Test Plan:
Make sure unity completes with no errors.
```lang=bash
% make unity
```
Reviewers: sdong, rven, igor, meyering
Reviewed By: igor, meyering
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D35385
Summary: Make it easier for people to run all the tests.
Test Plan: Run it.
Reviewers: rven, yhchiang, igor, MarkCallaghan, IslamAbdelRahman, igor.sugak, anthony, kradhakrishnan, meyering
Reviewed By: meyering
Subscribers: meyering, leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D35319
Summary:
Adds gtest fused source code into `third-party` directory. No manual changes.
gtest latest released 1.7 has clang dev compilation errors. Trunk version requires only one disabled warning (-Wno-missing-field-initializers)
Fused code is made as described here https://fburl.com/90806322
Details about why we need gtest source code instead of precompiled library https://fburl.com/90805763
Source used from http://googletest.googlecode.com/svn/trunk
Test Plan:
Build and notice no errors. Also check in logs that gtest-all.o being compiled gtest-all.o.
```lang=bash
% USE_CLANG=1 make all
```
Reviewers: lgalanis, yufei.zhu, rven, sdong, igor, meyering
Reviewed By: meyering
Subscribers: meyering, yhchiang, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D33345
Summary:
Here's my proposal for making our LOGs easier to read by machines.
The idea is to dump all events as JSON objects. JSON is easy to read by humans, but more importantly, it's easy to read by machines. That way, we can parse this, load into SQLite/mongo and then query or visualize.
I started with table_create and table_delete events, but if everybody agrees, I'll continue by adding more events (flush/compaction/etc etc)
Test Plan:
Ran db_bench. Observed:
2015/01/15-14:13:25.788019 1105ef000 EVENT_LOG_v1 {"time_micros": 1421360005788015, "event": "table_file_creation", "file_number": 12, "file_size": 1909699}
2015/01/15-14:13:25.956500 110740000 EVENT_LOG_v1 {"time_micros": 1421360005956498, "event": "table_file_deletion", "file_number": 12}
Reviewers: yhchiang, rven, dhruba, MarkCallaghan, lgalanis, sdong
Reviewed By: sdong
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D31647
Summary:
Any time one would modify a dependent of any *test*.cc file,
"make" would fail to rebuild the affected test binaries,
e.g., db_test. That was due to the fact that we deliberately
excluded those test-related files from the definition of SOURCES
and only $(SOURCES) was used to create the automatically-generated
.d dependency files. The fix is to generate a .d file for every
source file.
* src.mk: New file. Defines LIB_SOURCES, MOCK_SOURCES
and TEST_BENCH_SOURCES.
* Makefile: Include src.mk.
Reflect s/SOURCES/LIB_SOURCES/ renaming.
* build_tools/build_detect_platform: Remove the code
that was used to generate SOURCES= and MOCK_SOURCES=
definitions in make_config.mk. Those lists of files
are now hard-coded in src.mk. Hard-coding this list of
sources is desirable, because without that, one risks
including stray .cc files in a build. Not reproducible.
Test Plan:
Touch a file used by db_test's dependent .o files and ensure that
they are all recompiled. Before, none would be:
$ touch db/db_impl.h && make db_test
CC db/db_test.o
CC db/column_family.o
CC db/db_filesnapshot.o
CC db/db_impl.o
CC db/db_impl_debug.o
CC db/db_impl_readonly.o
CC db/forward_iterator.o
CC db/internal_stats.o
CC db/managed_iterator.o
CC db/repair.o
CC db/write_batch.o
CC utilities/compacted_db/compacted_db_impl.o
CC utilities/ttl/db_ttl_impl.o
CC util/ldb_cmd.o
CC util/ldb_tool.o
CC util/sst_dump_tool.o
CC util/xfunc.o
CCLD db_test
Reviewers: ljin, igor.sugak, igor, rven, sdong
Reviewed By: sdong
Subscribers: yhchiang, adamretter, fyrz, dhruba
Differential Revision: https://reviews.facebook.net/D33849
Summary:
This patch will update the Makefile and source code so that we can build RocksDB successfully on FreeBSD 10 and 11 (64-bit and 32-bit)
I have also encountered some problems when running tests on FreeBSD, I will try to fix them individually in different diffs
Notes:
- FreeBSD uses clang as it's default compiler (http://lists.freebsd.org/pipermail/freebsd-current/2012-September/036480.html)
- GNU C++ compiler have C++ 11 problems on FreeBSD (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193528)
- make is not gmake on FreeBSD (http://www.khmere.com/freebsd_book/html/ch01.html)
Test Plan:
Using VMWare Fusion Create 4 VM machines (FreeBSD 11 64-bit, FreeBSD 11 32-bit, FreeBSD 10 64-bit, FreeBSD 10 32-bit)
- pkg install git gmake gflags archivers/snappy
- git clone https://github.com/facebook/rocksdb.git
- apply this patch
- setenv CXX c++
- setenv CPATH /usr/local/include/
- setenv LIBRARY_PATH /usr/local/lib/
- gmake db_bench
- make sure compilation is successful and db_bench is running
- gmake all
- make sure compilation is successful
Reviewers: sdong, igor
Reviewed By: igor
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D33891
Summary: See https://www.facebook.com/groups/rocksdb.dev/permalink/718536351578215/. We should enable people to run without warning-as-error, since we break this so often for different compilers.
Test Plan: introduce an warning, compile sucessfully
Reviewers: yhchiang, meyering, igor.sugak, rven, sdong
Reviewed By: sdong
Subscribers: adamretter, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D33963
Summary:
* Updated Makefile to exit `make analyze` with status 1 if scan-build detected any bugs.
* scan-build automatically detects which c++ compiler to use, and some times is uses wrong ones (from $CPP). Added implicit parameters to use $CC and $CXX.
* Added `scan_build_report` directory to .gitignore file.
* Added `scan_build_report` directory to clean target.
Test Plan:
Run `make analyze` and verify that exit status is 1, if there are scan-build bugs detected.
Run `make clean` and verify that files in `scan_build_report` directory are deleted.
After running `make analyze; git status` and verify that no untracked files in `scan_build_report` directory.
Reviewers: meyering, sdong, igor
Reviewed By: igor
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D33831
Summary:
* Makefile (WARNING_FLAGS): Add -W and -Wextra, and at least for now,
-Wno-unused-parameter. Without that latter one, there would be
many errors/warnings about unused parameters.
Test Plan:
run these and ensure there is no warning:
make clean && make
make clean && USE_CLANG=1 make
Reviewers: ljin, sdong, igor.sugak, igor
Reviewed By: igor
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D33819
Summary:
Prior to this change, "make check" would always waste a lot of
time relinking 60+ binaries. With this change, it does that
only when the generated file, util/build_version.cc, changes,
and that happens only when the date changes or when the
current git SHA changes.
This change makes some other improvements: before, there was no
rule to build a deleted util/build_version.cc. If it was somehow
removed, any attempt to link a program would fail.
There is no longer any need for the separate file,
build_tools/build_detect_version. Its functionality is
now in the Makefile.
* Makefile (DEPFILES): Don't filter-out util/build_version.cc.
No need, and besides, removing that dependency was wrong.
(date, git_sha, gen_build_version): New helper variables.
(util/build_version.cc): New rule, to create this file
and update it only if it would contain new information.
* build_tools/build_detect_platform: Remove file.
* db/db_impl.cc: Now, print only date (not the time).
* util/build_version.h (rocksdb_build_compile_time): Remove
declaration. No longer used.
Test Plan:
- Run "make check" twice, and note that the second time no linking is performed.
- Remove util/build_version.cc and ensure that any "make"
command regenerates it before doing anything else.
- Run this: strings librocksdb.a|grep _build_.
That prints output including the following:
rocksdb_build_git_date:2015-02-19
rocksdb_build_git_sha:2.8.fb-1792-g3cb6cc0
Reviewers: ljin, sdong, igor
Reviewed By: igor
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D33591
Summary:
The "ar" command was excessively verbose.
This abbreviates it to be like CC and LD.
Also, factor "rs" options into ARFLAGS.
Test Plan:
$ rm librocksdb.a
$ make librocksdb.a
AR librocksdb.a
ar: creating librocksdb.a
Reviewers: ljin, sdong, igor.sugak, igor
Reviewed By: igor
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D33699
Summary:
Many link commands were identical.
Factor that out into a variable, AM_LINK, and use
it in place of all of those open-coded commands.
Test Plan: run "make check"
Reviewers: ljin, igor, sdong
Reviewed By: igor
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D33585
Summary:
With this change, make now prints a summary line for each
compiler and linker invocation, e.g.,:
CC db/builder.o
CC db/c.o
CC db/column_family.o
To see full commands, insert "V=1" into your make command.
E.g., run "make V=1 all" if you want it to print each command
in its full glory.
$^ is GNU make's abbreviation for the prerequisites of the current target.
These AM_V_... variables expand to some very short string like "CC" or
"LD", by default, so that the output of "make" is readable. If/when you
want more details, just build with "make V=1 ...", and make will print
each full command as it is executed. If you prefer to see the noise
all the time, and only want to optionally see the abbreviated output,
set AM_DEFAULT_VERBOSITY=1 in your environment, and then build with
V=0 to see the abbreviated command indicators.
Test Plan:
invoke make a few different ways and observe:
make clean; make # abbreviated
make clean; make V=0 # also abbreviated
make clean; make V=1 # full detail
Reviewers: sdong, ljin, igor
Reviewed By: igor
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D33579
Summary:
This is in preparation for some factorization.
* Makefile (deletefile_test): Add $(COVERAGEFLAGS) to link command.
(options_test): Remove explicit (redundant) dependency on
options_helper.o: that is already a dependent, via $(LIBOBJECTS)
Test Plan: run make
Reviewers: sdong, ljin, igor
Reviewed By: igor
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D33573
Summary:
There were Makefile rules to build those two targets,
but neither rule has worked for a long time, due to missing
dependent source files. Remove those rules.
Test Plan: run "make"
Reviewers: sdong, ljin, igor
Reviewed By: igor
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D33567
Summary:
Before this change, running "make" with no arguments would
silently run the rules for the "uninstall" target(!). Don't do that.
* Makefile (default): New, first target; depend on "all".
(uninstall, install): Do not hide the commands we run.
Test Plan:
Run "make" and verify that the rules for "uninstall" are no longer run.
Instead, note that many files are compiled and linked. Before, none were.
Reviewers: sdong, ljin, igor
Reviewed By: igor
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D33531
Summary:
* Makefile (dummy): Prefix this statement with "dummy := ",
so that it no longer triggers a syntax error from GNU make 3.80
and earlier. Reported by nielsl in
https://github.com/facebook/rocksdb/issues/509
Test Plan: run make
Reviewers: sdong, ljin, igor
Reviewed By: igor
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D33429
Summary:
The code being removed would invoke sed differently to work
around a portability difference in how sed -i works (different
on MacOS). Yet performing a host-type-based ifdef fails when
the tools installed do not match. That sed use was solely to
post-process the .d file. Instead, generate the desired output
directly, by using the compiler's -MT<FILE> option.
* Makefile (%.d: %.cc): With the prior use of Makefile-ifdef'd
sed, when building on MacOS with gnu sed, every run of this rule
would fail with a sed usage error. Also list each .d file as a
dependent.
Test Plan:
Ensure that a selected .d file is the same as before both with
g++ and with clang++. However, note that the new .d files each
contain a new reference to the .d file itself.
Reviewers: sdong, ljin, igor
Reviewed By: igor
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D33369
Summary:
Added new target ##make analyze## into Makefile. This command runs clang static analyzer and builds the sources as ##make all##. The result report is put into ##$(RocksDbSourceRoot)/can_build_report/##
If the development environment is a Facebook devserver and ##ROCKSDB_NO_FBCODE## is not set, then scan-build is used from fbcode. If it is run not on a Facebook devserver, scan-build should be available in ##$PATH##. I'll add details to wiki how to install scan-build on a non Facebook devserver environment.
Test Plan:
Run the fallowing commands on a Facebook devserver and Mac OS, and ensure no build or test errors.
```
% make all check -j32
% make clean
% USE_CLANG=1 make all -j32
% make analyze
% USE_CLANG=1 make analyze
```
Reviewers: sdong, lgalanis, leveldb, igor
Reviewed By: igor
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D32799
Summary: Add CappedFixTransform, which is the same as fixed length prefix extractor, except that when slice is shorter than the fixed length, it will use the full key.
Test Plan:
Add a test case for
db_test
options_test
and a new test
Reviewers: yhchiang, rven, igor
Reviewed By: igor
Subscribers: MarkCallaghan, leveldb, dhruba, yoshinorim
Differential Revision: https://reviews.facebook.net/D31887
Summary: This was a feature request by osquery. See task t5617758
Test Plan: compiles and memenv_test runs
Reviewers: yhchiang, rven, sdong
Reviewed By: sdong
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D32115
Summary:
There's a bug in TSAN (or libstdc++?) with std::shared_ptr<> for some reason. In db_test, only FlushSchedule is affected.
See more: https://groups.google.com/forum/#!topic/thread-sanitizer/vz_s-t226Vg
With this change and all other @sdong's and mine diffs, our db_test should be TSAN-clean. I'll move to other tests.
Test Plan: no more flush schedule when running TSAN
Reviewers: yhchiang, rven, sdong
Reviewed By: sdong
Subscribers: sdong, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D32469
Summary: We don't have plans to work on this in the short term. If we ever resurrect the project, we can find the code in the history. No need for it to linger around
Test Plan: no test
Reviewers: yhchiang, rven, sdong
Reviewed By: sdong
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D32349
Summary:
1. If WAL directory is different from db directory. Sync the directory after creating a log file under it.
2. After creating an SST file, sync its parent directory instead of DB directory.
3. change the check of kResetDeleteUnsyncedFiles in fault_injection_test. Since we changed the behavior to sync log files' parent directory after first WAL sync, instead of creating, kResetDeleteUnsyncedFiles will not guarantee to show post sync updates.
Test Plan: make all check
Reviewers: yhchiang, rven, igor
Reviewed By: igor
Subscribers: leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D32067
Summary: When you compile with COMPILE_WITH_TSAN=1, we will compile the code with -fsanitize=thread. This will resolve bunch of data race issues we might have.
Test Plan: COMPILE_WITH_TSAN=1 m db_test
Reviewers: yhchiang, rven, sdong
Reviewed By: sdong
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D32019
Summary:
This is a port of [[ https://github.com/google/leveldb/blob/master/db/fault_injection_test.cc | LevelDB's fault_injection_test ]] to RocksDB. Unfortunately it fails with:
```
==== Test FaultInjectionTest.FaultTest
db/fault_injection_test.cc:491: Corruption: no meta-nextfile entry in descriptor
#0 ./fault_injection_test() [0x41477a] rocksdb::FaultInjectionTest::PartialCompactTestReopenWithFault(rocksdb::FaultInjectionTest::ResetMethod, int, int) /data/users/tomdzk/rocksdb/db/fault_injection_test.cc:491
#1 ./fault_injection_test() [0x40a38a] rocksdb::_Test_FaultTest::_Run() /data/users/tomdzk/rocksdb/db/fault_injection_test.cc:517
#2 ./fault_injection_test() [0x415bea] rocksdb::_Test_FaultTest::_RunIt() /data/users/tomdzk/rocksdb/db/fault_injection_test.cc:507
#3 ./fault_injection_test() [0x584367] rocksdb::test::RunAllTests() /data/users/tomdzk/rocksdb/util/testharness.cc:70
#4 /usr/local/fbcode/gcc-4.8.1-glibc-2.17/lib/libc.so.6(__libc_start_main+0x10e) [0x7f7a40857efe] ?? ??:0
#5 ./fault_injection_test() [0x408bb8] _start ??:0
```
so I commented out the test invocation in the source code for now (lines 514-520) so it can be merged.
Test Plan: This is a new test.
Reviewers: igor
Reviewed By: igor
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D31587
Summary: I'm tired of double-tab when opening build_tools/<something>. This change will make bu<tab> fully complete my path :)
Test Plan: `vi bu<tab>` gives me `vi build_tools/` yay!
Reviewers: yhchiang, rven, sdong
Reviewed By: sdong
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D30639
Summary:
Create a benchmark for testing memtablereps. This diff is a bit rough, but it should do the trick until other bootcampers can clean it up.
Addressing comments
Removed the mutexes
Changed ReadWriteBenchmark to fix number of reads and count the number of writes we can perform in that time.
Test Plan:
Run it.
Below runs pass
./memtablerep_bench --benchmarks fillrandom,readrandom --memtablerep skiplist
./memtablerep_bench --benchmarks fillseq,readseq --memtablerep skiplist
./memtablerep_bench --benchmarks readwrite,seqreadwrite --memtablerep skiplist --num_operations 200 --num_threads 5
./memtablerep_bench --benchmarks fillrandom,readrandom --memtablerep hashskiplist
./memtablerep_bench --benchmarks fillseq,readseq --memtablerep hashskiplist
--num_scans 2
./memtablerep_bench --benchmarks fillseq,readseq --memtablerep vector
Reviewers: jpaton, ikabiljo, sdong
Reviewed By: sdong
Subscribers: dhruba, ameyag
Differential Revision: https://reviews.facebook.net/D22683
Summary:
Add support to allow nested config for block-based table factory. The format looks like this:
"write_buffer_size=1024;block_based_table_factory={block_size=4k};max_write_buffer_num=2"
Test Plan: unit test
Reviewers: yhchiang, rven, igor, ljin, jonahcohen
Reviewed By: jonahcohen
Subscribers: jonahcohen, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D29223
Summary: Add -fno-exceptions flag to ROCKSDB_LITE.
Test Plan: make OPT=-DROCKSDB_LITE shared_lib -j32
Reviewers: sdong, igor
Reviewed By: igor
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D29901
Summary:
While running rocksdb tests, we sometimes encounter errors and
the test run stops. We now provide a new make target call check_some
which restarts the test run from a specific test and continues from
there depending on the value of the environment variable ROCKSDBTESTS_START
Test Plan:
Run make check_some with different values of
ROCKSDBTESTS_START.
Reviewers: igor
Reviewed By: igor
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D29913
Summary:
Add GetThreadList API, which allows developer to track the
status of each process. Currently, calling GetThreadList will
only get the list of background threads in RocksDB with their
thread-id and thread-type (priority) set. Will add more support
on this in the later diffs.
ThreadStatus currently has the following properties:
// An unique ID for the thread.
const uint64_t thread_id;
// The type of the thread, it could be ROCKSDB_HIGH_PRIORITY,
// ROCKSDB_LOW_PRIORITY, and USER_THREAD
const ThreadType thread_type;
// The name of the DB instance where the thread is currently
// involved with. It would be set to empty string if the thread
// does not involve in any DB operation.
const std::string db_name;
// The name of the column family where the thread is currently
// It would be set to empty string if the thread does not involve
// in any column family.
const std::string cf_name;
// The event that the current thread is involved.
// It would be set to empty string if the information about event
// is not currently available.
Test Plan:
./thread_list_test
export ROCKSDB_TESTS=GetThreadList
./db_test
Reviewers: rven, igor, sdong, ljin
Reviewed By: ljin
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D25047
Currently maven publishing uses the library with debug symbols. What
leads to unnecessary big library sizes. Included strip to remove
unnecessary stuff. 40M -> 2.7M
Jim Meyering