NextGraph
/
rocksdb
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
1294 Commits
4 Branches
0 Tags
169 MiB
Tag: Branch: Tree: ad9a39c9b4
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'ad9a39c9b4'
${ noResults }
rocksdb/build_tools/fbcode.gcc471.sh

71 lines
3.2 KiB
Raw Normal View History Unescape

Compile leveldb with gcc 4.7.1 Test Plan: run unit tests Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5163
12 years ago
#!/bin/sh
#
# Set environment variables so that we can compile leveldb using
# fbcode settings. It uses the latest g++ compiler and also
# uses jemalloc
Dbid feature Summary: Create a new type of file on startup if it doesn't already exist called DBID. This will store a unique number generated from boost library's uuid header file. The use-case is to identify the case of a db losing all its data and coming back up either empty or from an image(backup/live replica's recovery) the key point to note is that DBID is not stored in a backup or db snapshot It's preferable to use Boost for uuid because: 1) A non-standard way of generating uuid is not good 2) /proc/sys/kernel/random/uuid generates a uuid but only on linux environments and the solution would not be clean 3) c++ doesn't have any direct way to get a uuid 4) Boost is a very good library that was already having linkage in rocksdb from third-party Note: I had to update the TOOLCHAIN_REV in build files to get latest verison of boost from third-party as the older version had a bug. I had to put Wno-uninitialized in Makefile because boost-1.51 has an unitialized variable and rocksdb would not comiple otherwise. Latet open-source for boost is 1.54 but is not there in third-party. I have notified the concerned people in fbcode about it. @kailiu : While releasing to third-party, an additional dependency will need to be created for boost in TARGETS file. I can help identify. Test Plan: Expand db_test to test 2 cases 1) Restarting db with Id file present - verify that no change to Id 2)Restarting db with Id file deleted - verify that a different Id is there after reopen Also run make all check Reviewers: dhruba, haobo, kailiu, sdong Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D13587
11 years ago
TOOLCHAIN_REV=fbe3b095a4cc4a3713730050d182b7b4a80c342f
Compile leveldb with gcc 4.7.1 Test Plan: run unit tests Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5163
12 years ago
TOOLCHAIN_EXECUTABLES="/mnt/gvfs/third-party/$TOOLCHAIN_REV/centos5.2-native"
TOOLCHAIN_LIB_BASE="/mnt/gvfs/third-party/$TOOLCHAIN_REV/gcc-4.7.1-glibc-2.14.1"
Dbid feature Summary: Create a new type of file on startup if it doesn't already exist called DBID. This will store a unique number generated from boost library's uuid header file. The use-case is to identify the case of a db losing all its data and coming back up either empty or from an image(backup/live replica's recovery) the key point to note is that DBID is not stored in a backup or db snapshot It's preferable to use Boost for uuid because: 1) A non-standard way of generating uuid is not good 2) /proc/sys/kernel/random/uuid generates a uuid but only on linux environments and the solution would not be clean 3) c++ doesn't have any direct way to get a uuid 4) Boost is a very good library that was already having linkage in rocksdb from third-party Note: I had to update the TOOLCHAIN_REV in build files to get latest verison of boost from third-party as the older version had a bug. I had to put Wno-uninitialized in Makefile because boost-1.51 has an unitialized variable and rocksdb would not comiple otherwise. Latet open-source for boost is 1.54 but is not there in third-party. I have notified the concerned people in fbcode about it. @kailiu : While releasing to third-party, an additional dependency will need to be created for boost in TARGETS file. I can help identify. Test Plan: Expand db_test to test 2 cases 1) Restarting db with Id file present - verify that no change to Id 2)Restarting db with Id file deleted - verify that a different Id is there after reopen Also run make all check Reviewers: dhruba, haobo, kailiu, sdong Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D13587
11 years ago
TOOL_JEMALLOC=jemalloc-3.3.1/9202ce3
Compile leveldb with gcc 4.7.1 Test Plan: run unit tests Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5163
12 years ago
# location of libhdfs libraries
if test "$USE_HDFS"; then
JAVA_HOME="/usr/local/jdk-6u22-64"
JINCLUDE="-I$JAVA_HOME/include -I$JAVA_HOME/include/linux"
GLIBC_RUNTIME_PATH="/usr/local/fbcode/gcc-4.7.1-glibc-2.14.1"
HDFSLIB=" -Wl,--no-whole-archive hdfs/libhdfs.a -L$JAVA_HOME/jre/lib/amd64 "
Conversion of db_bench, db_stress and db_repl_stress to use gflags Summary: Converted db_stress, db_repl_stress and db_bench to use gflags Test Plan: I tested by printing out all the flags from old and new versions. Tried defaults, + various combinations with "interesting flags". Also, tested by running db_crashtest.py and db_crashtest2.py. Reviewers: emayanke, dhruba, haobo, kailiu, sdong Reviewed By: emayanke CC: leveldb, xjin Differential Revision: https://reviews.facebook.net/D13581
11 years ago
HDFSLIB+=" -L$JAVA_HOME/jre/lib/amd64/server -L$GLIBC_RUNTIME_PATH/lib "
Compile leveldb with gcc 4.7.1 Test Plan: run unit tests Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5163
12 years ago
HDFSLIB+=" -ldl -lverify -ljava -ljvm "
fi
[fbcode] Also add glibc and libgcc includes Summary: We also need to use custom glibc and libgcc includes instead of system ones. Test Plan: 'make clean; make check'. Will also try on @dhruba's dev server. Reviewers: dhruba, kailiu Reviewed By: kailiu CC: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D14157
11 years ago
# location of libgcc
LIBGCC_INCLUDE=" -I $TOOLCHAIN_LIB_BASE/libgcc/libgcc-4.7.1/afc21dc/include"
LIBGCC_LIBS=" -L $TOOLCHAIN_LIB_BASE/libgcc/libgcc-4.7.1/afc21dc/libs"
# location of glibc
GLIBC_INCLUDE=" -I $TOOLCHAIN_LIB_BASE/glibc/glibc-2.14.1/99df8fc/include"
GLIBC_LIBS=" -L $TOOLCHAIN_LIB_BASE/glibc/glibc-2.14.1/99df8fc/lib"
Compile leveldb with gcc 4.7.1 Test Plan: run unit tests Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5163
12 years ago
# location of snappy headers and libraries
Fix ldb segfault and use static libsnappy for all builds Summary: Link statically against snappy, using the gvfs one for facebook environments, and the bundled one otherwise. In addition, fix a few minor segfaults in ldb when it couldn't open the database, and update .gitignore to include a few other build artifacts. Test Plan: make check Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D6855
12 years ago
SNAPPY_INCLUDE=" -I $TOOLCHAIN_LIB_BASE/snappy/snappy-1.0.3/7518bbe/include"
SNAPPY_LIBS=" $TOOLCHAIN_LIB_BASE/snappy/snappy-1.0.3/7518bbe/lib/libsnappy.a"
Compile leveldb with gcc 4.7.1 Test Plan: run unit tests Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5163
12 years ago
Add zlib to our builds and tweak histogram output Summary: $SUBJECT -- cosmetic fix for histograms, print P75/P99, and make sure zlib is enabled for our command line tools. Test Plan: compile, test db_bench with --compression_type=zlib Reviewers: heyongqiang Reviewed By: heyongqiang CC: adsharma, leveldb Differential Revision: https://reviews.facebook.net/D8445
12 years ago
# location of zlib headers and libraries
ZLIB_INCLUDE=" -I $TOOLCHAIN_LIB_BASE/zlib/zlib-1.2.5/91ddd43/include"
ZLIB_LIBS=" $TOOLCHAIN_LIB_BASE/zlib/zlib-1.2.5/91ddd43/lib/libz.a"
Move back to gcc4.7.1 Summary: Dhruba can't compile on gcc4.8.1 so I'm moving temporarily back to 4.7.1 until we figure out what's wrong with 4.8. on his server. Test Plan: It can compile on my devserver, but please 'arc patch' this diff and try compiling on your machine Reviewers: dhruba, kailiu Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14139
11 years ago
# location of bzip headers and libraries
BZIP_INCLUDE=" -I $TOOLCHAIN_LIB_BASE/bzip2/bzip2-1.0.6/91ddd43/include"
BZIP_LIBS=" $TOOLCHAIN_LIB_BASE/bzip2/bzip2-1.0.6/91ddd43/lib/libbz2.a"
Conversion of db_bench, db_stress and db_repl_stress to use gflags Summary: Converted db_stress, db_repl_stress and db_bench to use gflags Test Plan: I tested by printing out all the flags from old and new versions. Tried defaults, + various combinations with "interesting flags". Also, tested by running db_crashtest.py and db_crashtest2.py. Reviewers: emayanke, dhruba, haobo, kailiu, sdong Reviewed By: emayanke CC: leveldb, xjin Differential Revision: https://reviews.facebook.net/D13581
11 years ago
# location of gflags headers and libraries
GFLAGS_INCLUDE=" -I $TOOLCHAIN_LIB_BASE/gflags/gflags-1.6/91ddd43/include"
GFLAGS_LIBS=" $TOOLCHAIN_LIB_BASE/gflags/gflags-1.6/91ddd43/lib/libgflags.a"
Enable SSE when building with fbcode support. Summary: fbcode build now support SSE instructions. Delete older version of the compile-helper fbcode.sh. This is subsumed by fbcode.gcc471.sh. Test Plan: run make check Reviewers: heyongqiang, MarkCallaghan Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D6057
12 years ago
# use Intel SSE support for checksum calculations
Do not compile thrift for fbcode build. Summary: 1. The thrift libraries do not need to be built anyore. 2. SSE is dynamically detected via https://github.com/facebook/rocksdb/commit/1aae609b920f8cd4d93ac49798fa96367b9b864c Test Plan: compile and build Reviewers: sheki, emayanke Reviewed By: sheki CC: leveldb Differential Revision: https://reviews.facebook.net/D7665
12 years ago
export USE_SSE=" -msse -msse4.2 "
Enable SSE when building with fbcode support. Summary: fbcode build now support SSE instructions. Delete older version of the compile-helper fbcode.sh. This is subsumed by fbcode.gcc471.sh. Test Plan: run make check Reviewers: heyongqiang, MarkCallaghan Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D6057
12 years ago
Fix ldb segfault and use static libsnappy for all builds Summary: Link statically against snappy, using the gvfs one for facebook environments, and the bundled one otherwise. In addition, fix a few minor segfaults in ldb when it couldn't open the database, and update .gitignore to include a few other build artifacts. Test Plan: make check Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D6855
12 years ago
CC="$TOOLCHAIN_EXECUTABLES/gcc/gcc-4.7.1-glibc-2.14.1/bin/gcc"
Remove libevent Summary: We don't need that dependency Test Plan: make check Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14199
11 years ago
CXX="$TOOLCHAIN_EXECUTABLES/gcc/gcc-4.7.1-glibc-2.14.1/bin/g++ $JINCLUDE $SNAPPY_INCLUDE $ZLIB_INCLUDE $BZIP_INCLUDE $GFLAGS_INCLUDE"
Compile leveldb with gcc 4.7.1 Test Plan: run unit tests Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5163
12 years ago
AR=$TOOLCHAIN_EXECUTABLES/binutils/binutils-2.21.1/da39a3e/bin/ar
RANLIB=$TOOLCHAIN_EXECUTABLES/binutils/binutils-2.21.1/da39a3e/bin/ranlib
We should compile with -fPIC on non-fbcode environments also
11 years ago
CFLAGS="-B$TOOLCHAIN_EXECUTABLES/binutils/binutils-2.21.1/da39a3e/bin/gold -m64 -mtune=generic"
Use correct version of jemalloc. Summary: Use correct version of jemalloc. Test Plan: run unit tests Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5487
12 years ago
CFLAGS+=" -I $TOOLCHAIN_LIB_BASE/jemalloc/$TOOL_JEMALLOC/include -DHAVE_JEMALLOC"
[fbcode] Also add glibc and libgcc includes Summary: We also need to use custom glibc and libgcc includes instead of system ones. Test Plan: 'make clean; make check'. Will also try on @dhruba's dev server. Reviewers: dhruba, kailiu Reviewed By: kailiu CC: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D14157
11 years ago
CFLAGS+=" $LIBGCC_INCLUDE $GLIBC_INCLUDE"
Add -DROCKSDB_FALLOCATE_PRESENT to fbcode build
11 years ago
CFLAGS+=" -DROCKSDB_PLATFORM_POSIX -DROCKSDB_ATOMIC_PRESENT -DROCKSDB_FALLOCATE_PRESENT"
Move back to gcc4.7.1 Summary: Dhruba can't compile on gcc4.8.1 so I'm moving temporarily back to 4.7.1 until we figure out what's wrong with 4.8. on his server. Test Plan: It can compile on my devserver, but please 'arc patch' this diff and try compiling on your machine Reviewers: dhruba, kailiu Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14139
11 years ago
CFLAGS+=" -DSNAPPY -DGFLAGS -DZLIB -DBZIP2"
Compile leveldb with gcc 4.7.1 Test Plan: run unit tests Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5163
12 years ago
Add libevent include and lib directories Summary: Without this fix, I see failures like this: [zshao@dev1049 /data/users/zshao/rocksdb] . fbcode.gcc471.sh; gmake clean libleveldb.a . . . ./thrift/lib/cpp/async/TEventUtil.h:22:32: fatal error: event.h: No such file or directory Test Plan: . fbcode.gcc471.sh; make clean libleveldb.a Reviewers: dhruba, emayanke, sheki Reviewed By: sheki CC: leveldb Differential Revision: https://reviews.facebook.net/D7497
12 years ago
EXEC_LDFLAGS=" -Wl,--whole-archive $TOOLCHAIN_LIB_BASE/jemalloc/$TOOL_JEMALLOC/lib/libjemalloc.a"
Dbid feature Summary: Create a new type of file on startup if it doesn't already exist called DBID. This will store a unique number generated from boost library's uuid header file. The use-case is to identify the case of a db losing all its data and coming back up either empty or from an image(backup/live replica's recovery) the key point to note is that DBID is not stored in a backup or db snapshot It's preferable to use Boost for uuid because: 1) A non-standard way of generating uuid is not good 2) /proc/sys/kernel/random/uuid generates a uuid but only on linux environments and the solution would not be clean 3) c++ doesn't have any direct way to get a uuid 4) Boost is a very good library that was already having linkage in rocksdb from third-party Note: I had to update the TOOLCHAIN_REV in build files to get latest verison of boost from third-party as the older version had a bug. I had to put Wno-uninitialized in Makefile because boost-1.51 has an unitialized variable and rocksdb would not comiple otherwise. Latet open-source for boost is 1.54 but is not there in third-party. I have notified the concerned people in fbcode about it. @kailiu : While releasing to third-party, an additional dependency will need to be created for boost in TARGETS file. I can help identify. Test Plan: Expand db_test to test 2 cases 1) Restarting db with Id file present - verify that no change to Id 2)Restarting db with Id file deleted - verify that a different Id is there after reopen Also run make all check Reviewers: dhruba, haobo, kailiu, sdong Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D13587
11 years ago
EXEC_LDFLAGS+=" -Wl,--no-whole-archive $TOOLCHAIN_LIB_BASE/libunwind/libunwind-1.0.1/350336c/lib/libunwind.a"
Remove libevent Summary: We don't need that dependency Test Plan: make check Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14199
11 years ago
EXEC_LDFLAGS+=" $HDFSLIB $SNAPPY_LIBS $ZLIB_LIBS $BZIP_LIBS $GFLAGS_LIBS"
Various build cleanups/improvements Summary: Specific changes: 1) Turn on -Werror so all warnings are errors 2) Fix some warnings the above now complains about 3) Add proper dependency support so changing a .h file forces a .c file to rebuild 4) Automatically use fbcode gcc on any internal machine rather than whatever system compiler is laying around 5) Fix jemalloc to once again be used in the builds (seemed like it wasn't being?) 6) Fix issue where 'git' would fail in build_detect_version because of LD_LIBRARY_PATH being set in the third-party build system Test Plan: make, make check, make clean, touch a header file, make sure rebuild is expected Reviewers: dhruba Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D7887
12 years ago
[fbcode] Also add glibc and libgcc includes Summary: We also need to use custom glibc and libgcc includes instead of system ones. Test Plan: 'make clean; make check'. Will also try on @dhruba's dev server. Reviewers: dhruba, kailiu Reviewed By: kailiu CC: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D14157
11 years ago
PLATFORM_LDFLAGS="$LIBGCC_LIBS $GLIBC_LIBS "
Add optional clang compile mode Summary: clang is an alternate compiler based on llvm. It produces nicer error messages and finds some bugs that gcc doesn't, such as the size_t change in this file (which caused some write return values to be misinterpreted!) Clang isn't the default; to try it, do "USE_CLANG=1 make" or "export USE_CLANG=1" then make as normal Test Plan: "make check" and "USE_CLANG=1 make check" Reviewers: dhruba Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D7899
12 years ago
Move back to gcc4.7.1 Summary: Dhruba can't compile on gcc4.8.1 so I'm moving temporarily back to 4.7.1 until we figure out what's wrong with 4.8. on his server. Test Plan: It can compile on my devserver, but please 'arc patch' this diff and try compiling on your machine Reviewers: dhruba, kailiu Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14139
11 years ago
EXEC_LDFLAGS_SHARED="$SNAPPY_LIBS $ZLIB_LIBS $BZIP_LIBS $GFLAGS_LIBS"
Compile leveldb with gcc 4.7.1 Test Plan: run unit tests Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5163
12 years ago
Moving VALGRIND_VER which takes the valgrind version from third party to fbcode.gcc471.sh file Summary: the valgrind version being used is in facebook specific path and should be moved to the fbcode.gcc471.sh file instead of the makefile. The execution takes the environment's default valgrind version if the fbcode.gcc471.sh's valgrind_version is not available. Test Plan: make valgrind_check Reviewers: dhruba, sheki, akushner Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D9213
11 years ago
VALGRIND_VER="$TOOLCHAIN_LIB_BASE/valgrind/valgrind-3.8.1/91ddd43/bin/"
Remove snappy from RocksDB distribution Summary: Argumentation here: https://github.com/facebook/rocksdb/issues/9 Even though we include snappy in the distribution, we do not link with it if we don't have snappy installed on the system. Installing snappy is easy nowadays, just type: sudo apt-get install libsnappy-dev Test Plan: compile on ubuntu Reviewers: dhruba, kailiu Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14133
11 years ago
export CC CXX AR RANLIB CFLAGS EXEC_LDFLAGS EXEC_LDFLAGS_SHARED VALGRIND_VER