f10e5a40a3
Enables more Clippy lints
...
Auto-enable all Clippy lints that are not in nursery but a blacklist
1 year ago
001b6e07b7
Enforces "return self not must use" lint
1 year ago
76deca135c
BulkLoader: Uses thread::scope
1 year ago
7175784356
Improves Clippy lint list
2 years ago
1d02098b70
Fix missing GSPO compaction
2 years ago
6af6c9c0eb
Bulk loader: fixes the progress indicator display
2 years ago
7787be6e84
Bulk loader: pre-allocate some buffers
2 years ago
13976014e7
Bulk loader: reduces default system parameters
...
Uses by default targets of 1GB of RAM and 2 threads.
Data parsing is in most of the case slower than ingestion so no more than 2 threads are used anyway.
2 years ago
fbcbd60c0e
Python: adds Store.bulk_extend
2 years ago
26f4e2dc98
Allows to have secondary instances in memory
2 years ago
855c39146d
Replaces Store::open_with_options with explicit variants
...
Trades enums and structs for methods
2 years ago
aa9476b9cc
Add option to open rocksdb in secondary mode
...
The database can be opened once in primary mode, but may be opened
multiple times in secondary mode. It's not possible to write data to the
database in secondary mode.
Secondary mode does not support checkpoints so there might be data
inconsistencies when data is changed while said data is queried. It
might happen that the result data mixes both elements before and after
the change that can not happen in primary mode.
2 years ago
1fa0633db3
Library: adds basic WASI support
...
Adds tests for WASM+JS and WASI to the CI
2 years ago
808c9db007
String formatting: uses inline syntax
2 years ago
59aea75a30
Uses try_into for slice to array conversions
2 years ago
4cea628570
Upgrades to sysinfo 0.26
2 years ago
95fd439dc2
Updates dependencies
2 years ago
ad710dc825
Bulk loader: removes dead code
...
There were some dead code and key-values where deduplicated twice
3 years ago
cc4e3f8b0f
Bulk loader: avoids to write duplicated values
3 years ago
3cbde014a6
Limits the bulk loader thread count to 4
...
Allows writing bigger files
3 years ago
d0b3d76bf1
Bulk loader: use as much memory as possible
3 years ago
54c6f2c19f
Bulk load: use megabytes as unit
3 years ago
9e969cd8eb
Adds methods to configure bulk loader system consumption
3 years ago
adea1899e5
Bulk loader: rewrite API and add hook to display progress indicators
3 years ago
ac64b58825
Fixes wasm build
3 years ago
822dd60596
Fixes invalid database generated from bulk load
...
Adds an invariant validation test
3 years ago
203bd4c080
Adds a backup system
...
RocksDB provides a great cheap backup feature thanks to the immutable SST storage.
Sadly it is not compatible with in-memory databases which do not rely on the SST files.
3 years ago
a91ea89fff
Moves str2id outside of transactions
...
Allows avoiding conflicts and reducing transactional guarantees
3 years ago
a33dbb6d06
Avoids abusing std::io::Error
...
Introduces new specific errors and make them implement Into<io::Error> for compatibility
3 years ago
44d1a5f04c
Makes bulk load partial insertions atomic
...
Ensures that the stores does not end up in an inconsistent state.
3 years ago
8d20f65890
Adds back in memory WASM support
3 years ago
ed17e86853
Renames write_stt_files to insert_stt_files
3 years ago
a3e40556da
Automatically retries transactions on failure
3 years ago
eb4fb8254a
Lock the graph name when clearing a named graph
...
Makes sure we conflict if a quad is inserted in the graph while we clear it
3 years ago
e99d6f4ad0
Makes bulk loader multithreaded
3 years ago
b9d4f912dc
Bulk load: avoid not needed lookup
...
It's fine to have some extra inserts and avoid reading the DB state
3 years ago
7bb41e655e
SPARQL: we do not need to ensure strings exist anymore
3 years ago
e59c4612b2
RocksDB: Removes merge and compact operators
3 years ago
b7ee3a6767
Removes string usage counting
3 years ago
6f44a5956b
Simplifies bulk load API
3 years ago
7b1c4e0ad5
Returns a clean error on transaction read after commit
3 years ago
607aa0b0dd
WIP: Transactions
3 years ago
9f414c13fd
Simplifies RocksDB SST API
3 years ago
196d6d6576
Adds bulk load by directly write RocksDB SSTs
3 years ago
569000b5ea
Storage: adds write batch
...
Atomic insertions and huge performance improvements for RocksDB
3 years ago
986d3e60bb
RocksDB: bulk load
...
TODO: we do a lot of lookups during load for GC, so the option is not very useful
3 years ago
98f9a307b8
Exposes a function to compact the database
...
Use it after load
3 years ago
00ad6e634f
Cleaner storage backend architecture
3 years ago
73cd761229
Removes unused TermEncoder
3 years ago
0ee25d7579
RocksDB: Adds prefix bloom filters
3 years ago
Tpt