The error was:
util/logging.cc: In function 'int rocksdb::AppendHumanMicros(uint64_t, char*, int)':
error: util/logging.cc:41:39: integer overflow in expression [-Werror=overflow]
} else if (micros < 1000000l * 60 * 60) {
^
error: util/logging.cc:41:39: comparison between signed and unsigned integer expressions [-Werror=sign-compare]
Summary: It's useful to know if we have compression support or no
Test Plan:
Observed this in my LOG:
2015/03/26-10:34:35.460681 7f5b322b7840 Snappy supported
2015/03/26-10:34:35.460682 7f5b322b7840 Zlib supported
2015/03/26-10:34:35.460686 7f5b322b7840 Bzip supported
2015/03/26-10:34:35.460687 7f5b322b7840 LZ4 NOT supported
Reviewers: sdong, yhchiang
Reviewed By: yhchiang
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D35955
[maa@srv2-nskb-devg2 rocksdb-master]$ CXX=/usr/local/CC/gcc-4.7.4/bin/g++ EXTRA_CXXFLAGS=-std=c++11 DISABLE_WARNING_AS_ERROR=1 make db_bench
CC db/db_bench.o
db/db_bench.cc: In member function 'rocksdb::Slice rocksdb::Benchmark::AllocateKey(std::unique_ptr<const char []>*)':
db/db_bench.cc:1434:41: error: use of deleted function 'void std::unique_ptr<_Tp [], _Dp>::reset(_Up) [with _Up = char*; _Tp = const char; _Dp = std::default_delete<const char []>]'
In file included from /usr/local/CC/gcc-4.7.4/lib/gcc/x86_64-unknown-linux-gnu/4.7.4/../../../../include/c++/4.7.4/memory:86:0,
from ./include/rocksdb/db.h:14,
from ./db/dbformat.h:14,
from ./db/db_impl.h:21,
from db/db_bench.cc:33:
Summary:
* Makefile (COMPILE_WITH_TSAN): Avoid a link failure by disabling
-pg when building with TSAN enabled.
Now that "make check" builds all $(PROGRAMS), it is linking
a few programs that were not normally linked before.
For example, this would fail to link with the following diagnostic:
COMPILE_WITH_TSAN=1 make -j40 log_and_apply_bench
CCLD log_and_apply_bench
ld: /usr/lib/../lib64/gcrt1.o: relocation R_X86_64_32S against `__libc_csu_fini' can not be used when making a shared object; recompile with -fPIC
/usr/lib/../lib64/gcrt1.o: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
Makefile:511: recipe for target 'log_and_apply_bench' failed
make: *** [log_and_apply_bench] Error 1
Since removing -pg is sufficient to get past this link
failure, and no one cares about profiling TSAN-enabled
binaries anyway, we will refrain from linking with -pg
when TSAN testing is enabled. Use a new variable, "pg"
which is set to "-pg" in most cases, but that is made
empty when COMPILE_WITH_TSAN is set.
Test Plan:
Now, this succeeds:
rm -f log_and_apply_bench
COMPILE_WITH_TSAN=1 make -j40 log_and_apply_bench
Reviewers: igor.sugak, rven, sdong, ljin, igor
Reviewed By: igor
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D35943
Summary:
* Makefile (check): Cause "make check" to build all $(PROGRAMS),
so that it verifies that the few benchmark-only source files that
not already built via "make check" do compile and link successfully.
Test Plan:
run "make clean; make check", and verify that
table/table_reader_bench.cc is now compiled.
Before, it was not, which led to an incomplete fix
for a build break.
Reviewers: ljin, igor.sugak, rven, igor, sdong
Reviewed By: sdong
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D35883
Summary: Update HISTORY.md for 3.10.0
Test Plan: no code chagne.
Reviewers: sdong, rven, igor
Reviewed By: igor
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D35871
Summary: Fixing issues with get context function.
Test Plan: Run make commit-prereq
Reviewers: sdong, meyering, yhchiang
Reviewed By: sdong
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D35853
Summary: Fix compile error when NROCKSDB_THREAD_STATUS is not used.
Test Plan: make dbg OPT=-DNROCKSDB_THREAD_STATUS -j32
Reviewers: sdong, igor, rven
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D35847
Summary:
We have addded new stats and perf_context for measuring the merge and filter operation time consumption.
We have bounded all the merge operations within the GUARD statment and collected the total time for these operations in the DB.
Test Plan: WIP
Reviewers: rven, yhchiang, kradhakrishnan, igor, sdong
Reviewed By: sdong
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D34377
Summary:
Report elapsed time of a thread operation in micros in ThreadStatus
instead of start time of a thread operation in seconds since the
Epoch, 1970-01-01 00:00:00 (UTC).
Test Plan:
./db_bench --benchmarks=fillrandom --num=100000 --threads=40 \
--max_background_compactions=10 --max_background_flushes=3 \
--thread_status_per_interval=1000 --key_size=16 --value_size=1000 \
--num_column_families=10
Sample Output:
ThreadID ThreadType cfName Operation ElapsedTime Stage State
140667724562496 High Pri column_family_name_000002 Flush 772.419 ms FlushJob::WriteLevel0Table
140667728756800 High Pri default Flush 617.845 ms FlushJob::WriteLevel0Table
140667732951104 High Pri column_family_name_000005 Flush 772.078 ms FlushJob::WriteLevel0Table
140667875557440 Low Pri column_family_name_000008 Compaction 1409.216 ms CompactionJob::Install
140667737145408 Low Pri
140667749728320 Low Pri
140667816837184 Low Pri column_family_name_000007 Compaction 1071.815 ms CompactionJob::ProcessKeyValueCompaction
140667787477056 Low Pri column_family_name_000009 Compaction 772.516 ms CompactionJob::ProcessKeyValueCompaction
140667741339712 Low Pri
140667758116928 Low Pri column_family_name_000004 Compaction 620.739 ms CompactionJob::ProcessKeyValueCompaction
140667753922624 Low Pri
140667842003008 Low Pri column_family_name_000006 Compaction 1260.079 ms CompactionJob::ProcessKeyValueCompaction
140667745534016 Low Pri
Reviewers: sdong, igor, rven
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D35769
Summary: There is no alternative to GetLiveFiles() function
Test Plan: none
Reviewers: sdong
Reviewed By: sdong
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D35805
Summary:
Improve ThreadStatusSingleCompaction in two ways:
1. Use SYNC_POINT to ensure compaction won't happen
before the test finishes its "Put Phase" instead of
using sleep.
2. In Put Phase, it continues until we have sufficient
number of L0 files. Note that during the put phase,
there won't be any compaction that consumes L0 files
because of item 1.
Test Plan: ./db_test --gtest_filter="*ThreadStatusSingleCompaction*"
Reviewers: sdong, igor, rven
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D35727
Summary:
Limiting verbose printing to "command=scan"
Test Plan:
Run make check and manual testing of sst_dump_test
Reviewers: sdong
CC: leveldb
Task ID: #6575982
Blame Rev:
Summary:
In 3.10 the C++ code was extended with a MemEnv implementation. This
is now also available in RocksJava.
Changes:
- Extraced abstract super class Env
- Introduced RocksMemEnv
- Remove unnecessary disposeInternal method. The disposal of the default environment is managed by C++ so there needs to be no disposeInternal method in Java.
- Introduced a RocksMemEnvTest, which is aligned with the C++ equivalent.
Test Plan:
make rocksdbjava
make jtest
Reviewers: adamretter, yhchiang, ankgup87
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D35619
Summary:
RocksDB offers the possibility to set different compression types
on a per level basis. This shall be also available using RocksJava.
Test Plan:
make rocksdbjava
make jtest
Reviewers: adamretter, yhchiang, ankgup87
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D35577
Summary: DBTest doesn't clean up wal directory. It might cause failure after a failure test run. Fix it.
Test Plan:
Run unit tests
Try open DB with non-empty db_path/wal.
Reviewers: rven, yhchiang, kradhakrishnan, igor
Reviewed By: igor
Subscribers: leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D35559
Summary: Modified rocksdb status assertions ASSERT_OK and EXPECT_OK to print error message from Status::ToString() when failed.
Test Plan: Modify a test to fail status assertions ASSERT_OK and EXPECT_OK and notice an error message that came from Status::ToString()
Reviewers: meyering, sdong, yhchiang, igor
Reviewed By: igor
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D35469
Summary: Cleaning util/testharness.h
Test Plan:
Make completes with no errors.
```
% make all
```
Reviewers: meyering, igor, sdong
Reviewed By: sdong
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D35487
Summary:
To understand the bug read t5943287 and check out the new test in column_family_test (ReadDroppedColumnFamily), iter 0.
RocksDB contract allowes you to read a drop column family as long as there is a live reference. However, since our iteration ignores dropped column families, AddLiveFiles() didn't mark files of a dropped column families as live. So we deleted them.
In this patch I no longer ignore dropped column families in the iteration. I think this behavior was confusing and it also led to this bug. Now if an iterator client wants to ignore dropped column families, he needs to do it explicitly.
Test Plan: Added a new unit test that is failing on master. Unit test succeeds now.
Reviewers: sdong, rven, yhchiang
Reviewed By: yhchiang
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D32535
Summary: Deleted some redundant code. More comming.
Test Plan:
```lang=bash
% USE_CLANG=1 make check
```
Reviewers: meyering, sdong, igor
Reviewed By: igor
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D35463
Summary:
Before this change overflowing size_t values led to a checked Exception.
After that change:
size_t overflows on 32-Bit architecture throw now an IllegalArgumentException,
which removes the necessity for a developer to catch these Exceptions explicitly.
This is especially an advantage for developers targeting 64-Bit systems because
it is not necessary anymore to catch exceptions which are never thrown on a 64-Bit
system.
Test Plan:
make clean jclean rocksdbjava jtest
mvn -f rocksjni.pom package
Reviewers: adamretter, yhchiang, ankgup87
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D34923
Summary:
changed make unity target to use $LIB_SOURCES as a source of library source code. In the old way (using find) table/mock_table.h was added to a list of library objects and this was a course of `make unity` break.
`build_tools/unity` contains some redundant code, I deleted it and moved the functionality in the Makefile.
Test Plan:
Make sure unity completes with no errors.
```lang=bash
% make unity
```
Reviewers: sdong, rven, igor, meyering
Reviewed By: igor, meyering
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D35385
Summary: Suprisingly, the only way we use this vector is to keep track of level0 compactions. Thus, I simplified it.
Test Plan: make check
Reviewers: rven, yhchiang, sdong
Reviewed By: sdong
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D35313
Summary: This is a simple change to make db_test::MultiThreadedDBTest as value parameterized test. There is a value of creating a separate set of such tests later.
Test Plan:
```lang=bash
% make db_test
% ./make db_test
```
Also with the following command I can execute all db_test in 2:37.87 on my box
```
% ./db_test --gtest_list_tests | sed 's/\# GetParam.*//' | tr -d ' ' | env time parallel --gnu --eta --joblog=LOG -- 'TEST_TMPDIR=/dev/shm/rocksdb-{} ./db_test --gtest_filter="*{}"'
```
Reviewers: igor, rven, meyering, sdong
Reviewed By: meyering
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D35361
Summary: Make it easier for people to run all the tests.
Test Plan: Run it.
Reviewers: rven, yhchiang, igor, MarkCallaghan, IslamAbdelRahman, igor.sugak, anthony, kradhakrishnan, meyering
Reviewed By: meyering
Subscribers: meyering, leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D35319
Summary: Add a DB property for number of deletions in memtables. It can sometimes help people debug slowness because of too many deletes.
Test Plan: Add test cases.
Reviewers: rven, yhchiang, kradhakrishnan, igor
Reviewed By: igor
Subscribers: leveldb, dhruba, yoshinorim
Differential Revision: https://reviews.facebook.net/D35247
Summary:
table_cache_remove_scan_count_limit is marked as deprecated in RocksDB C++.
(see rocksdb/options.h). This patch made necessary changes on RocksJava
side.
Test Plan:
make rocksdbjava -j32
make jtest
Reviewers: rven, igor, fyrz, adamretter, ankgup87, sdong
Reviewed By: sdong
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D35355
xiaoxichen