On Windows two tests fail that use MockTable:
flush_job_test and compaction_job_test with the following message:
compaction_job_test_je.exe : Assertion failed: result.size() == 4,
file c:\dev\rocksdb\rocksdb\table\mock_table.cc, line 110
Investigation reveals that this failure occurs when a 4 byte
ID written to a beginning of the physically open file (main
contents remains in a in-memory map) can not be read back.
The reason for the failure is that the ID is written directly
to a WritableFile bypassing WritableFileWriter. The side effect of that
is that pending_sync_ never becomes true so the file is never flushed,
however, the direct cause of the failure is that the filesize_ member
of the WritableFileWriter remains zero. At Close() the file is truncated
to that size and the file becomes empty so the ID can not be read back.
Summary: IO Posix depends on too many .h files. Move most of them to .cc files.
Test Plan: make all
Reviewers: anthony, rven, IslamAbdelRahman, yhchiang, kradhakrishnan, igor
Reviewed By: igor
Subscribers: leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D49479
Summary: Update rocksdb-lego-determinator to include running make check under ROCKSDB_LITE
Test Plan: will be tested after landing in fbcode
Reviewers: sdong, yhchiang, igor, kradhakrishnan
Reviewed By: kradhakrishnan
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D49065
Summary: include/posix/io_posix.h doesn't not prevent multiple includes. Need to fix it. It is also breaking unity build.
Test Plan: Run unity build and see error go away.
Reviewers: rven, igor, IslamAbdelRahman, kradhakrishnan, anthony
Subscribers: leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D49281
Summary: This patch splits the posix storage backend into Env and
the actual *File implementations. The motivation is to allow other Envs
to use posix as a library. This enables a storage backend different from
posix to split its secondary storage between a normal file system
partition managed by posix, and it own media.
Test Plan: No new functionality is added to posix Env or the library,
thus the current tests should suffice.
Summary: Need to pass through the memtable parameter.
Test Plan: built, tested through myrocks
Reviewers: igor, sdong, rven
Reviewed By: rven
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D49167
Summary:
in tools/db_crashtest.py, cmd_params['db'] by default is a lambda expression, not the actual db_name.
fix by get the db_name before passing it to gen_cmd.
Test Plan: run `make crashtest`
Reviewers: sdong
Reviewed By: sdong
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D49119
Summary: Use DEBUG_LEVEL=0 in make release and make clean
Test Plan:
make clean
make release -j32
Reviewers: MarkCallaghan, sdong, anthony, IslamAbdelRahman, igor
Reviewed By: igor
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D49125
Test code errors are currently blocking Windows Release builew
We do not want spend time building in Release what we can not run
We want to eliminate a source of most frequent errors when people
check-in test only code which can not be built in Release.
This feature will work only if you invoke msbuild against rocksdb.sln
Invoking it against ALL_BUILD target will attempt to build everything.
* add validation is git found
* add validation is .git folder exists in project (project zip archive download without .git folder)
* get head commit SHA if git found and .git folder exists
Tested:
* configure project by CMake 3.0.0 successfully (with and without git), with project zip archive (without .git folder) and with project cloned from github
* configure project by command: cmake -G "Visual Studio 12 Win64"
* build solution by Visual Studio
* manually validate that file utils/build_version.cc contains valid head revision value
Summary: Manual compaction should not fill block cache. Add the verification in unit test
Test Plan: Run the test
Reviewers: yhchiang, kradhakrishnan, rven, IslamAbdelRahman, anthony, igor
Reviewed By: igor
Subscribers: leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D49089
Summary: Adding the ability to upload logs and db content to storage after the
completion of the job
Test Plan: Manual run
Reviewers:
CC: leveldb@
Task ID: #8754201
Blame Rev:
Summary: Before this diff `arc lint` on non-fb machine issued warnings. Now it doesn't.
Test Plan: `arc lint` is quiet.
Reviewers: yhchiang, sdong
Reviewed By: sdong
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D49071
Summary: crash_test sometimes fails, hitting the add file overlapping assert. Add information in info logs help us to find the bug.
Test Plan: Run all test suites. Do some manual tests to make sure printing is correct.
Reviewers: kradhakrishnan, yhchiang, anthony, IslamAbdelRahman, rven, igor
Reviewed By: igor
Subscribers: leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D49017
Summary: Bump the version to 4.2 ( the unreleased version ), so that when fbcode_unittests run it can differentiate between old and new APIs
Test Plan: make check
Reviewers: sdong, yhchiang, igor
Reviewed By: igor
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D49041
Introduce new tags for records that have a log_number. This changes the
header size from 7 to 11 for these records, making this a
backward-incompatible change.
If we read a record that belongs to a different log_number (i.e., a
previous instantiation of this log file, before it was most recently
recycled), we return kOldRecord from ReadPhysicalRecord. ReadRecord
will translate this into a kEof or kBadRecord depending on what the
WAL recovery mode is.
We make several adjustments to the log_test.cc tests to compensate for the
fact that the header size varies between the two modes.
Signed-off-by: Sage Weil <sage@redhat.com>
Summary: We don't yet have a CI build for iOS, so our iOS compile gets broken sometimes. Most of the errors are from assumption that size_t is 64-bit, while it's actually 32-bit on some (all?) iOS platforms. This diff fixes the compile.
Test Plan:
TARGET_OS=IOS make static_lib
Observe there are no warnings
Reviewers: sdong, anthony, IslamAbdelRahman, kradhakrishnan, yhchiang
Reviewed By: yhchiang
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D49029
Summary:
merge tools/db_crashtest2.py into tools/db_crashtest.py
python tools/db_crashtest.py -h # show help message, ALL parameters can be overwrite by arguments
Example usages:
python tools/db_crashtest.py blackbox # run blackbox with default parameters
python tools/db_crashtest.py blackbox --simple
python tools/db_crashtest.py whitebox # run whitebox with default parameters
python tools/db_crashtest.py whitebox --simple
all default parameters are identical to previous version.
Test Plan: `make crash_test` and make sure it can run with expected parameters pased to db_stress.
Reviewers: igor, rven, anthony, IslamAbdelRahman, yhchiang, sdong
Reviewed By: sdong
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D48567
Summary: Remove warning about an issue that was resolved. Turns out the issue was a false-alarm.
Test Plan: n/a
Reviewers: igor, yhchiang, rven, sdong
Reviewed By: sdong
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D49011
Summary:
In the current make file, DEBUG_LEVEL is forced to set to 1
instead of default to 1. This patch fix this issue.
Test Plan:
DEBUG_LEVEL=2 make db_bench -j32
DEBUG_LEVEL=0 make db_bench -j32
DEBUG_LEVEL=0 make release -j32
And see whether there's a warning pops up.
Reviewers: MarkCallaghan, igor
Reviewed By: igor
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D48933
Summary: As above.
Test Plan: USE_CLANG=1 make check -j
Reviewers: igor
Reviewed By: igor
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D48981
Move the WAL recovery mode logic out of ReadPhysicalRecord. To do this we
introduce a new type indicating when we fail to read a valid header.
Signed-off-by: Sage Weil <sage@redhat.com>
We will need the log number to validate the recycle-style CRCs. The log
is helpful for debugging, but optional, as not all callers have it.
Signed-off-by: Sage Weil <sage@redhat.com>
When we recycle log files, we need to mix the log number into the CRC
for each record. Note that for logs that don't get recycled (like the
manifest), we always pass a log_number of 0 and false.
Signed-off-by: Sage Weil <sage@redhat.com>
If log recycling is enabled, put old WAL files on a recycle queue instead of
deleting them. When we need a new log file, take a recycled file off the
list if one is available.
Signed-off-by: Sage Weil <sage@redhat.com>
Add an environment method to reuse an existing file. Provide a generic
implementation that does a simple rename + open (writeable), and also a
posix variant that is more careful about error handling (if we fail to
open, do not rename, etc.).
Signed-off-by: Sage Weil <sage@redhat.com>
Summary: In MyRocks, it is sometimes important to get propeties only for the subset of the database. This diff implements the API in RocksDB.
Test Plan: ran the GetPropertiesOfTablesInRange
Reviewers: rven, sdong
Reviewed By: sdong
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D48651
Summary:
Add rocksdb.num-running-compactions and rocksdb.num-running-flushes
to GetIntProperty() that reports the number of currently running
compactions / flushes.
Test Plan: augmented existing tests in db_test
Reviewers: igor, IslamAbdelRahman, sdong
Reviewed By: sdong
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D48693
Summary:
Add kill points in:
1. after creating a file
2. before writing a manifest record
3. before syncing manifest
4. before creating a new current file
5. after creating a new current file
Test Plan: Run all current tests.
Reviewers: yhchiang, igor, anthony, IslamAbdelRahman, rven, kradhakrishnan
Reviewed By: kradhakrishnan
Subscribers: leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D48855
Summary:
We are cleaning up dependencies.
This diff takes a first step at moving memtable files to their own
directory called memtable. In future diffs, we will move other memtable
files from db to memtable.
Test Plan: make check
Reviewers: sdong
Reviewed By: sdong
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D48915
Summary:
MyRocks needs the ability to clear a snapshot for Read Committed support
Test Plan: transaction_test
Reviewers: anthony
Reviewed By: anthony
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D48861
Summary:
For half of the crash_test run, disable fail point for file appending, in order to trigger other fail point more frequently.
Also, tune crash test parameter a little bit for it to initialize faster.
Test Plan: Run crash_test and make sure it issues db_stress commands as expected.
Reviewers: yhchiang, igor, rven, anthony, IslamAbdelRahman, kradhakrishnan
Reviewed By: kradhakrishnan
Subscribers: leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D48843
Dmitri Smirnov