rocksdb

fork of https://github.com/oxigraph/rocksdb and https://github.com/facebook/rocksdb for nextgraph and oxigraph

Changyu Bi f24ef5d6ab Fix BackupEngineTest.ExcludeFiles memory leak (#11066 ) Summary: Valgrind was complaining about the test BackupEngineTest.ExcludeFiles. The cause is backup_engine not being freed similar to https://github.com/facebook/rocksdb/issues/9610. ``` ==18228== Command: ./backup_engine_test --gtest_filter=BackupEngineTest.ExcludeFiles ==18228== Note: Google Test filter = BackupEngineTest.ExcludeFiles [==========] Running 1 test from 1 test case. [----------] Global test environment set-up. [----------] 1 test from BackupEngineTest [ RUN ] BackupEngineTest.ExcludeFiles [ OK ] BackupEngineTest.ExcludeFiles (16264 ms) [----------] 1 test from BackupEngineTest (16273 ms total) [----------] Global test environment tear-down [==========] 1 test from 1 test case ran. (16306 ms total) [ PASSED ] 1 test. ==18228== ==18228== HEAP SUMMARY: ==18228== in use at exit: 14,099 bytes in 159 blocks ==18228== total heap usage: 255,328 allocs, 255,169 frees, 497,538,546 bytes allocated ==18228== ==18228== 19 bytes in 1 blocks are possibly lost in loss record 4 of 67 ==18228== at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so) ==18228== by 0x1E752D: void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char>(char, char, std::forward_iterator_tag) [clone .constprop.0] (basic_string.tcc:219) ==18228== by 0x1F1898: _M_construct_aux<char> (basic_string.h:251) ==18228== by 0x1F1898: _M_construct<char> (basic_string.h:270) ==18228== by 0x1F1898: basic_string (basic_string.h:455) ==18228== by 0x1F1898: construct<std::__cxx11::basic_string<char>, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&> (new_allocator.h:146) ==18228== by 0x1F1898: construct<std::__cxx11::basic_string<char>, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&> (alloc_traits.h:483) ==18228== by 0x1F1898: push_back (stl_vector.h:1189) ==18228== by 0x1F1898: rocksdb::(anonymous namespace)::TestFs::NewWritableFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::FileOptions const&, std::unique_ptr<rocksdb::FSWritableFile, std::default_delete<rocksdb::FSWritableFile> >, rocksdb::IODebugContext) (backup_engine_test.cc:208) ==18228== by 0x4B3583: rocksdb::NewWritableFile(rocksdb::FileSystem, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::unique_ptr<rocksdb::FSWritableFile, std::default_delete<rocksdb::FSWritableFile> >, rocksdb::FileOptions const&) (read_write_util.cc:23) ==18228== by 0x31C3A8: rocksdb::DBImpl::CreateWAL(unsigned long, unsigned long, unsigned long, rocksdb::log::Writer) (db_impl_open.cc:1752) ==18228== by 0x321A8C: rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle, std::allocator<rocksdb::ColumnFamilyHandle> >, rocksdb::DB, bool, bool) (db_impl_open.cc:1852) ==18228== by 0x322E7F: Open (db_impl_open.cc:1660) ==18228== by 0x322E7F: rocksdb::DB::Open(rocksdb::Options const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::DB) (db_impl_open.cc:1637) ==18228== by 0x1EE1CD: InitializeDBAndBackupEngine (backup_engine_test.cc:724) ==18228== by 0x1EE1CD: rocksdb::(anonymous namespace)::BackupEngineTest::OpenDBAndBackupEngine(bool, bool, rocksdb::(anonymous namespace)::BackupEngineTest::ShareOption) (backup_engine_test.cc:732) ==18228== by 0x217585: rocksdb::(anonymous namespace)::BackupEngineTest_ExcludeFiles_Test::TestBody() (backup_engine_test.cc:4232) ==18228== by 0x296143: HandleSehExceptionsInMethodIfSupported<testing::Test, void> (gtest-all.cc:3899) ==18228== by 0x296143: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test, void (testing::Test::)(), char const*) (gtest-all.cc:3935) ==18228== by 0x28A0A5: testing::Test::Run() [clone .part.0] (gtest-all.cc:3973) ==18228== by 0x28A364: Run (gtest-all.cc:3965) ==18228== by 0x28A364: testing::TestInfo::Run() [clone .part.0] (gtest-all.cc:4149) ... ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/11066 Test Plan: make -j24 J=24 ROCKSDBTESTS_SUBSET=backup_engine_test valgrind_check_some Reviewed By: ajkr Differential Revision: D42297791 Pulled By: cbi42 fbshipit-source-id: db67982b27b91cc78e1a9f4a96da0cba7c9785b7		1 year ago
.circleci	Upgrade CircleCI Windows Build (#10090 )	2 years ago
.github/workflows	ci: add GitHub token permissions for workflow (#10549 )	2 years ago
buckifier	Enable BLACK for internal_repo_rocksdb (#10710 )	2 years ago
build_tools	Print stack traces on frozen tests in CI (#10828 )	2 years ago
cache	Add a SecondaryCache::InsertSaved() API, use in CacheDumper impl (#10945 )	1 year ago
cmake	gcc-11 and cmake related cleanup (#9286 )	2 years ago
coverage	Enable BLACK for internal_repo_rocksdb (#10710 )	2 years ago
db	Fix typo in flushing stats CF (#11055 )	1 year ago
db_stress_tool	Prevent db_stress failure when io_uring is disabled (#11045 )	1 year ago
docs	Bump nokogiri from 1.13.9 to 1.13.10 in /docs (#11024 )	1 year ago
env	Added placeholders for MADV defines (#10881 )	2 years ago
examples	Add rocksdb_backup_restore_example to examples/.gitignore (#10825 )	2 years ago
file	Fix async prefetch heap use after free (#11049 )	1 year ago
fuzz	Add some missing headers (#10519 )	2 years ago
include/rocksdb	add c-api for setting option optimize_filters_for_memory (#11044 )	1 year ago
java	Add missing range conflict check between file ingestion and RefitLevel() (#10988 )	1 year ago
logging	Observe and warn about misconfigured HyperClockCache (#10965 )	1 year ago
memory	Fix use of make_unique in Arena::AllocateNewBlock (#11012 )	1 year ago
memtable	Run clang format against files under example/, memory/ and memtable/ folders (#10893 )	2 years ago
microbench	Avoid allocations/copies for large `GetMergeOperands()` results (#10458 )	2 years ago
monitoring	Add an unittest for Periodic compaction conflict with ongoing compaction (#10908 )	1 year ago
options	Ignore max_compaction_bytes for compaction input that are within output key-range (#10835 )	2 years ago
plugin	Add initial CMake support to plugin (#9214 )	2 years ago
port	Fix include of windows.h in mmap.h (#10885 )	2 years ago
table	Avoid mixing sync and async prefetch (#11050 )	1 year ago
test_util	~SleepingBackgroundTask() to wake up the sleeping task (#11036 )	1 year ago
third-party	Meta-internal folly integration with F14FastMap (#9546 )	2 years ago
tools	Revert PR 10777 "Fix FIFO causing overlapping seqnos in L0 files due to overla…" (#10999 )	1 year ago
trace_replay	fix compile warnings (#10976 )	1 year ago
util	Improve error messages for SST footer and size errors (#11009 )	1 year ago
utilities	Fix BackupEngineTest.ExcludeFiles memory leak (#11066 )	1 year ago
.clang-format	A script that automatically reformat affected lines	10 years ago
.gitignore	Git ignore .clangd/ (#10817 )	2 years ago
.lgtm.yml	Create lgtm.yml for LGTM.com C/C++ analysis (#4058 )	6 years ago
.watchmanconfig	Added .watchmanconfig file to rocksdb repo (#5593 )	5 years ago
AUTHORS	Update RocksDB Authors File	7 years ago
CMakeLists.txt	Add ability to have unit tests for ROCKSDB_PLUGINS (#11052 )	1 year ago
CODE_OF_CONDUCT.md	Adopt Contributor Covenant	5 years ago
CONTRIBUTING.md	Add Code of Conduct	6 years ago
COPYING	Add GPLv2 as an alternative license.	7 years ago
DEFAULT_OPTIONS_HISTORY.md	Add Options::DisableExtraChecks, clarify force_consistency_checks (#9363 )	2 years ago
DUMP_FORMAT.md	First version of rocksdb_dump and rocksdb_undump.	9 years ago
HISTORY.md	Add missing range conflict check between file ingestion and RefitLevel() (#10988 )	1 year ago
INSTALL.md	Update supported VS versions in INSTALL.md (#9823 )	2 years ago
LANGUAGE-BINDINGS.md	Add grocksdb in Go language bindings (#10498 )	2 years ago
LICENSE.Apache	Change RocksDB License	7 years ago
LICENSE.leveldb	Add back the LevelDB license file	7 years ago
Makefile	Add ability to have unit tests for ROCKSDB_PLUGINS (#11052 )	1 year ago
PLUGINS.md	Add pmem-rocksdb-plugin link in PLUGINs.md (#9934 )	2 years ago
README.md	Remove Travis CI (#10407 )	2 years ago
ROCKSDB_LITE.md	Fix remaining uses of "backupable" (#9792 )	2 years ago
TARGETS	Consider range tombstone in compaction output file cutting (#10802 )	1 year ago
USERS.md	Add Apache Spark as a user (#10993 )	1 year ago
Vagrantfile	Adding CentOS 7 Vagrantfile & build script	6 years ago
WINDOWS_PORT.md	Update branch name in WINDOWS_PORT.md (#8745 )	3 years ago
common.mk	Clean up variables for temporary directory (#9961 )	2 years ago
crash_test.mk	Allow a custom DB cleanup command to be passed to db_crashtest.py (#10883 )	2 years ago
issue_template.md	Add Google Group to Issue Template	4 years ago
rocksdb.pc.in	build: fix pkg-config file generation (#9953 )	2 years ago
src.mk	Consider range tombstone in compaction output file cutting (#10802 )	1 year ago
thirdparty.inc	Fix build jemalloc api (#5470 )	5 years ago

README.md

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/main/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.

Questions and discussions are welcome on the RocksDB Developers Public Facebook group and email list on Google Groups.

License

RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.