Summary:
We forgot to add the new flag in internal build script. Add it.
Closes https://github.com/facebook/rocksdb/pull/2360
Differential Revision: D5121428
Pulled By: siying
fbshipit-source-id: af72d48cd855b37df1ce3c1fbb00c80377ba6e4f
Summary:
We've had a couple CockroachDB users fail to build RocksDB on exotic platforms, so I figured I'd try my hand at solving these issues upstream. The problems stem from a) `USE_SSE=1` being too aggressive about turning on SSE4.2, even on toolchains that don't support SSE4.2 and b) RocksDB attempting to detect support for thread-local storage based on OS, even though it can vary by compiler on the same OS.
See the individual commit messages for details. Regarding SSE support, this PR should change virtually nothing for non-CMake based builds. `make`, `PORTABLE=1 make`, `USE_SSE=1 make`, and `PORTABLE=1 USE_SSE=1 make` function exactly as before, except that SSE support will be automatically disabled when a simple SSE4.2-using test program fails to compile, as it does on OpenBSD. (OpenBSD's ports GCC supports SSE4.2, but its binutils do not, so `__SSE_4_2__` is defined but an SSE4.2-using program will fail to assemble.) A warning is emitted in this case. The CMake build is modified to support the same set of options, except that `USE_SSE` is spelled `FORCE_SSE42` because `USE_SSE` is rather useless now that we can automatically detect SSE support, and I figure changing options in the CMake build is less disruptive than changing the non-CMake build.
I've tested these changes on all the platforms I can get my hands on (macOS, Windows MSVC, Windows MinGW, and OpenBSD) and it all works splendidly. Let me know if there's anything you object to—I obviously don't mean to break any of your build pipelines in the process of fixing ours downstream.
Closes https://github.com/facebook/rocksdb/pull/2199
Differential Revision: D5054042
Pulled By: yiwu-arbug
fbshipit-source-id: 938e1fc665c049c02ae15698e1409155b8e72171
Summary:
Currently, the RPM package will install the lib and header files into `/usr/package/lib` and `/usr/package/include` which is not in the default search paths. It is reasonable to install them under `/usr/lib` and `/usr/include` so that no extra configuration is required.
Closes https://github.com/facebook/rocksdb/pull/2221
Differential Revision: D5054030
Pulled By: yiwu-arbug
fbshipit-source-id: 1d23de5ff21f07e6738c9dfa04429acd7a839143
Summary:
- added a feature test in build_detect_platform to check whether sched_getcpu() is available. glibc offers it only on some platforms (e.g., linux but not mac); this way should be easier than maintaining a list of platforms on which it's available.
- refactored PhysicalCoreID() to be simpler / less repetitive. ordered the conditional compilation clauses from most-to-least preferred
Closes https://github.com/facebook/rocksdb/pull/2272
Differential Revision: D5038093
Pulled By: ajkr
fbshipit-source-id: 81d7db3cc620250de220bdeb3194b2b3d7673de7
Summary:
VALGRIND_VER was left empty after moving the environment to GCC-5. Set it back.
Closes https://github.com/facebook/rocksdb/pull/2234
Differential Revision: D4978534
Pulled By: siying
fbshipit-source-id: f0640d58e8f575f75fb3f8b92e686c9e0b6a59bb
Summary:
Replacement of #2147
The change was squashed due to a lot of conflicts.
Closes https://github.com/facebook/rocksdb/pull/2194
Differential Revision: D4929799
Pulled By: siying
fbshipit-source-id: 5cd49c254737a1d5ac13f3c035f128e86524c581
Summary:
When people are working off of a rocksdb fork, i.e. when their 'origin'
points to github.com/<username>/rocksdb, the script creates a new branch
and pushes to their origin. The new branch created by this script should
instead be pushed to github.com/facebook/rocksdb. Many people might
have named facebook/rocksdb remote as 'upstream' (or something else).
This fix provides an option to specify the remote to push the branch to.
The default is still 'origin'
More context:
When I created 5.4 branch using this script, it got pushed to sagar0/rocksdb instead of facebook/rocksdb, as I was working off of a fork. My 'origin' was pointing to sagar0/rocksdb. My 'upstream' was set to 'facebook/rocksdb'. So, I had to manually push the branch to my 'upstream'.
Closes https://github.com/facebook/rocksdb/pull/2156
Differential Revision: D4885333
Pulled By: sagar0
fbshipit-source-id: 9410eab5bd9bbefc340059800bd6b8434406729d
Summary:
Rework test running script.
New options SuiteRun - runs specified executables as google suite
test cases in parallel.
Run - this option now runs executables in parallel the same as 'tests'
RunAll - scans for test executables and attempts to run them all
as suites except those mentiones in RunOnly (hardcoded in the script)
or specified either in $ExcludeTestCases $ExcludeTestExe
Closes https://github.com/facebook/rocksdb/pull/2089
Differential Revision: D4832212
Pulled By: yiwu-arbug
fbshipit-source-id: 954990c
Summary:
Since non_shn CI was made to run in parallel, /dev/shm is automatically used. It defeated the purpose of the test to cover a non-ramfs file system.
Closes https://github.com/facebook/rocksdb/pull/2031
Differential Revision: D4764804
Pulled By: siying
fbshipit-source-id: 5666bda
Summary:
This fixes an issue when the most recent readers assume that alignment is always set even if direct io is off.
Also adjust slightly appveyor script to run db_basic_test cases concurrently.
Closes https://github.com/facebook/rocksdb/pull/1959
Differential Revision: D4671972
Pulled By: IslamAbdelRahman
fbshipit-source-id: 1886620
Summary:
The PATH update should put the Java path in the beginning, rather than the end. Otherwise, it will be overwritten. Also upgrade the Java version.
Closes https://github.com/facebook/rocksdb/pull/1912
Differential Revision: D4609854
Pulled By: siying
fbshipit-source-id: 3dc04f2
Summary:
Remove disableDataSync, and another similarly named disable_data_sync options.
This is being done to simplify options, and also because the performance gains of this feature can be achieved by other methods.
Closes https://github.com/facebook/rocksdb/pull/1859
Differential Revision: D4541292
Pulled By: sagar0
fbshipit-source-id: 5b3a6ca
Summary:
The gcc-4.8.1-glibc-2.17 platform is deprecated and will be removed
soon.
Closes https://github.com/facebook/rocksdb/pull/1839
Differential Revision: D4509684
Pulled By: siying
fbshipit-source-id: 3efe296
Summary:
Update clang-format script to format diff since last commit from master,
instead of just last commit. In our common workflow we usually endup
with multiple commits for a single PR. This change make it easier to
format all stacking changes.
Closes https://github.com/facebook/rocksdb/pull/1684
Differential Revision: D4340597
Pulled By: yiwu-arbug
fbshipit-source-id: c18949e
Summary:
Previously:
$ make format
Makefile:104: Warning: Compiling in debug mode. Don't use the resulting binary in production
build_tools/format-diff.sh
You didn't have clang-format-diff.py available in your computer!
You can download it by running:
curl http://goo.gl/iUW1u2
Makefile:868: recipe for target 'format' failed
make: *** [format] Error 128
$ curl http://goo.gl/iUW1u2 > ~/bin/clang-format-diff.py
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 276 0 276 0 0 148 0 --:--:-- 0:00:01 --:--:-- 148m
$ more ~/bin/clang-format-diff.py
<HTML>
<HEAD>
<TITLE>Moved Permanently</TITLE>
</HEAD>
<BODY BGCOLOR="#FFFFFF" TEXT="#000000">
<H1>Moved Permanently</H1>
The document has moved <A HREF="https://raw.github.com/leaningtech/duetto-clang/master/tools/clang-format/clang-format-diff.py">here</A>.
</BODY>
</HTML>
Closes https://github.com/facebook/rocksdb/pull/1680
Differential Revision: D4338495
Pulled By: yiwu-arbug
fbshipit-source-id: e2b24d8
Summary:
This diff includes an implementation of CompactionFilter that allows
users to write CompactionFilter in Lua. With this ability, users can
dynamically change compaction filter logic without requiring building
the rocksdb binary and restarting the database.
To compile, WITH_LUA_PATH must be specified to the base directory
of lua.
Closes https://github.com/facebook/rocksdb/pull/1478
Differential Revision: D4150138
Pulled By: yhchiang
fbshipit-source-id: ed84222
Summary:
Rocksdb currently has many references to std::map.emplace_back()
which is not implemented in gcc 4.7, but valid in gcc 4.8. Confirmed that
it did not build with gcc 4.7, but builds fine with gcc 4.8
Closes https://github.com/facebook/rocksdb/pull/1272
Differential Revision: D4101385
Pulled By: IslamAbdelRahman
fbshipit-source-id: f6af453
Summary:
valgrind_test is the correct way to run valgrind tests.
this is becasue we need to force DISABLE_JEMALLOC
Test Plan: Running sandcastle and contrun
Reviewers: IslamAbdelRahman
Reviewed By: IslamAbdelRahman
Subscribers: andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D65451
Summary:
ZSTD 1.0.0 is coming. We can finally add a support of ZSTD without worrying about compatibility.
Still keep ZSTDNotFinal for compatibility reason.
Test Plan: Run all tests. Run db_bench with ZSTD version with RocksDB built with ZSTD 1.0 and older.
Reviewers: andrewkr, IslamAbdelRahman
Reviewed By: IslamAbdelRahman
Subscribers: cyan, igor, IslamAbdelRahman, leveldb, andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D63141
Summary: Splitting the makefile part of D55581.
Test Plan:
make all check -j32
ROCKSDB_FBCODE_BUILD_WITH_481=1 make all check -j32
ROCKSDB_NO_FBCODE=1 make all check -j32
export TBB_BASE=/mnt/gvfs/third-party2/tbb/afa54b33cfcf93f1d90a3160cdb894d6d63d5dca/4.0_update2/gcc-4.9-glibc-2.20/e9936bf;
ROCKSDB_NO_FBCODE=1 CFLAGS="-I $TBB_BASE/include" LDFLAGS="-L $TBB_BASE/lib -Wl,-rpath=$TBB_BASE/lib" make all check -j32
Reviewers: IslamAbdelRahman, sdong
Reviewed By: sdong
Subscribers: kradhakrishnan, yhchiang, IslamAbdelRahman, andrewkr, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D56979
Summary:
We have replaced LEVELDB_PLATFORM_POSIX with ROCKSDB_PLATFORM_POSIX in our code
replace it in Makefile
Test Plan: make check
Reviewers: sdong, yiwu, andrewkr, lightmark
Reviewed By: lightmark
Subscribers: andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D61365
Summary:
We may not have permission on all /dev/shm to fix the sticky bit.
Making the sticky bit fix advisory.
Test Plan: Run CI job locally
Subscribers: andrewkr, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D61371
Summary: We rely on a basic evaluation logic: when using an expression `A || B`, the `B` part will only get evaluated when `A` fails. Therefore the task creation tool is guaranteed to run if previous build/test step failed. To indicate the correct return value to shell, the task creation tool will call `exit(1)` which will cause Sandcastle to mark it as a failure.
Test Plan:
- Land the changes.
- Trigger a RocksDB contrun - observe the results.
- Look at the results from the nightly runs and fix issues as necessary.
Current testing so far has been in isolation for various components.
Reviewers: andrewkr, IslamAbdelRahman, sdong
Reviewed By: sdong
Subscribers: andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D60909
Summary:
Removing moreutils from sandcastle and adding gnu parallel.
Then passing in J= nproc command
Test Plan: Testing on sandcastle
Reviewers: sdong, kradhakrishnan
Reviewed By: kradhakrishnan
Subscribers: andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D61017
Summary: Increase test timeout to some tests to unblock CI.
Test Plan: watch how it runs.
Reviewers: kradhakrishnan, andrewkr, IslamAbdelRahman
Reviewed By: IslamAbdelRahman
Subscribers: IslamAbdelRahman, leveldb, andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D61263
Summary:
Tsan crash white-box test was disabled because it never ends. Re-enable it with reduced killing odds.
Add a parameter in crash test script to allow we pass it through an environment variable.
Test Plan: Run it manually.
Reviewers: IslamAbdelRahman
Reviewed By: IslamAbdelRahman
Subscribers: leveldb, andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D61053
Summary: RocksDB behavior is slightly different between data on tmpfs and normal file systems. Add a test case to run RocksDB on normal file system.
Test Plan: See the tests launched by Phabricator
Reviewers: kradhakrishnan, IslamAbdelRahman, gunnarku
Reviewed By: gunnarku
Subscribers: leveldb, andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D60963
Summary: I tried on my host and TSAN black box test runs well. I didn't see any problem with white box crash test too but it chance of hitting crash point is too low that it may run almost forever. First re-enable black box crash test to unblock the job.
Test Plan: Run it locally.
Reviewers: kradhakrishnan, andrewkr, IslamAbdelRahman
Reviewed By: IslamAbdelRahman
Subscribers: leveldb, andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D60951
Summary: As Title.
Test Plan: See how the diff works.
Reviewers: kradhakrishnan, andrewkr, gunnarku
Reviewed By: gunnarku
Subscribers: leveldb, andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D60933
Summary: To make sure that we'll have additional verification for release builds, define a new category and add `make release` to per-diff/post-commit tests. This should in theory prevent the release MyRocks integration builds breaks from happening.
Test Plan:
- `[p]arc diff --preview`
- Observe the execution in Sandcastle and make sure that release build and tests are executed.
Reviewers: sdong
Reviewed By: sdong
Subscribers: andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D60441
Summary: Fix two minor typos and update the file name which is used to trigger the runs in case new changes have been committed.
Test Plan: - Testing with a private Sandcastle instance.
Reviewers: sdong, mung
Reviewed By: mung
Subscribers: andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D59919
Summary:
The main PHP code churn is caused by extracting the common code from `FacebookArcanistConfiguration.php` and `FacebookOldArcanistConfiguration.php` into `RocksDBCommonDeterminator.php`. This is necessary both for reducing the duplication of code and making sure that we can execute the common core logic separately from continuous runs.
The main logic in `RocksDBCommonDeterminator.php` remains quite the same with the exception of some things:
- Adding separation between the cases when a diff is submitted //vs.// when the code is triggered from a continuous run. There are certain actions which we should do in a case of diff only.
- Adding reporting - now the person who authored the diff will receive e-mail notifications if any of the jobs have failed.
- Enabling assertions and making sure that we'll terminate on failure. This is an internal code used by competent engineers, so instead of `if (!condition) { echo "Something"; exit(1); }` for every invariant I think that `assert(condition)` provides better readability with the same behavior. Especially taking into account that we're talking about things which shouldn't ever happen.
Enabling this entire process will be triggered internally and will be a subject of a separate code review. We should discuss the details of triggering continuous RocksDB build and tests on that diff.
Test Plan:
- Make sure that `[p]arc diff` scenario isn't broken by verifying that tests validating this diff will pass.
- Private testing of triggering the continuous build script.
- Once the changes will land then author an internal job which will use the script and verify its validity.
Reviewers: sdong, yhchiang, kradhakrishnan
Reviewed By: kradhakrishnan
Subscribers: andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D59811
Summary: current libgcc lib path is broken, update it and make sure the new one exists
Test Plan:
verify that the new path exists
make check -j64
Reviewers: yiwu, sdong, andrewkr
Reviewed By: andrewkr
Subscribers: andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D59535
hyunwoo