9e969cd8eb
Adds methods to configure bulk loader system consumption
3 years ago
adea1899e5
Bulk loader: rewrite API and add hook to display progress indicators
3 years ago
ac64b58825
Fixes wasm build
3 years ago
822dd60596
Fixes invalid database generated from bulk load
...
Adds an invariant validation test
3 years ago
203bd4c080
Adds a backup system
...
RocksDB provides a great cheap backup feature thanks to the immutable SST storage.
Sadly it is not compatible with in-memory databases which do not rely on the SST files.
3 years ago
a91ea89fff
Moves str2id outside of transactions
...
Allows avoiding conflicts and reducing transactional guarantees
3 years ago
a33dbb6d06
Avoids abusing std::io::Error
...
Introduces new specific errors and make them implement Into<io::Error> for compatibility
3 years ago
44d1a5f04c
Makes bulk load partial insertions atomic
...
Ensures that the stores does not end up in an inconsistent state.
3 years ago
8d20f65890
Adds back in memory WASM support
3 years ago
ed17e86853
Renames write_stt_files to insert_stt_files
3 years ago
a3e40556da
Automatically retries transactions on failure
3 years ago
eb4fb8254a
Lock the graph name when clearing a named graph
...
Makes sure we conflict if a quad is inserted in the graph while we clear it
3 years ago
e99d6f4ad0
Makes bulk loader multithreaded
3 years ago
b9d4f912dc
Bulk load: avoid not needed lookup
...
It's fine to have some extra inserts and avoid reading the DB state
3 years ago
7bb41e655e
SPARQL: we do not need to ensure strings exist anymore
3 years ago
e59c4612b2
RocksDB: Removes merge and compact operators
3 years ago
b7ee3a6767
Removes string usage counting
3 years ago
6f44a5956b
Simplifies bulk load API
3 years ago
7b1c4e0ad5
Returns a clean error on transaction read after commit
3 years ago
607aa0b0dd
WIP: Transactions
3 years ago
9f414c13fd
Simplifies RocksDB SST API
3 years ago
196d6d6576
Adds bulk load by directly write RocksDB SSTs
3 years ago
569000b5ea
Storage: adds write batch
...
Atomic insertions and huge performance improvements for RocksDB
3 years ago
986d3e60bb
RocksDB: bulk load
...
TODO: we do a lot of lookups during load for GC, so the option is not very useful
3 years ago
98f9a307b8
Exposes a function to compact the database
...
Use it after load
3 years ago
00ad6e634f
Cleaner storage backend architecture
3 years ago
73cd761229
Removes unused TermEncoder
3 years ago
0ee25d7579
RocksDB: Adds prefix bloom filters
3 years ago
821cac9c4c
Sets some RocksDB options doing perf improvements
3 years ago
5da9fd4f14
Fixes Storage::contains_str
...
Takes care of the counter if not a key might be removed after the next compaction even if assumed to be contained
3 years ago
a7a0b7bbf3
String garbage collector: do not decrease counters set to MAX.
...
We do not keep track of counting beyond i32::MAX
3 years ago
cddb5900ef
Storage: Manages strings GC using merge operation
...
Removes old strings during compaction
3 years ago
bf0f178c41
Adds a ColumnFamilyDefinition struct to store column family options
3 years ago
def7a3ce72
RocksDB: Adds low priority writes
...
Allows RocksDB to defer garbage collection if useful
3 years ago
dfefe6cd1c
Uses transactional RocksDB
...
First stop to real transaction support
3 years ago
e297de73d1
Refactors backend code
...
Makes it easier to implement transactions
3 years ago
0bd512a14c
Avoids using API without column families
3 years ago
cb146efd7c
Use directly RocksDB C API
...
Will allow more customizations in the future
3 years ago
063683088d
Moves back from Sled to RocksDB
...
Simple implementation without transactions
3 years ago
97bb227042
Migrates to Rust 2021
3 years ago
e9608fb2eb
Adds wasm32 support to lib crate
...
The storage layer is in memory and not optimized yet
4 years ago
f084cfe332
Enables more Clippy lints and enforces them
4 years ago
1abda73dc9
String garbage collector
4 years ago
8a1683eba2
Fixes named graphs insertion
4 years ago
8606877e33
Simplifies term encoding code
4 years ago
fa7ae0353f
Makes Storage edit method take the explicit term
...
Allows to abstract away the insertion process from SPARQL
Adds also some optimized methods for SPARQL UPDATE operations
4 years ago
c18281183a
Uses std::io::Result where relevant
4 years ago
e2bbc29198
Makes encoded term not copy
4 years ago
68aab2563c
Simplifies string storage traits
4 years ago
0efc5b4654
Renames SledStore to Store and cleans up files hierarchy
4 years ago
Tpt