1d02098b70
Fix missing GSPO compaction
2 years ago
a977adff91
Allows to collect RocksDB statistics
...
Hidden behind the "rocksdb_debug" features
2 years ago
6af6c9c0eb
Bulk loader: fixes the progress indicator display
2 years ago
7787be6e84
Bulk loader: pre-allocate some buffers
2 years ago
13976014e7
Bulk loader: reduces default system parameters
...
Uses by default targets of 1GB of RAM and 2 threads.
Data parsing is in most of the case slower than ingestion so no more than 2 threads are used anyway.
2 years ago
fbcbd60c0e
Python: adds Store.bulk_extend
2 years ago
26f4e2dc98
Allows to have secondary instances in memory
2 years ago
5f68cb3746
Cleans up RocksDB wrapper code and implements backup from read-only and secondary instances
2 years ago
b2385509a6
Improves read only and secondary test coverage
...
Adds the missing get method
2 years ago
855c39146d
Replaces Store::open_with_options with explicit variants
...
Trades enums and structs for methods
2 years ago
f7637ee5a5
Add option to open database in read-only mode
...
With read-only it's not possible to modify the data. Updates to the data
are possible via a primary instance of oxigraph, but will not be
reflected. The data is frozen at the time the read-only server is
started.
2 years ago
aa9476b9cc
Add option to open rocksdb in secondary mode
...
The database can be opened once in primary mode, but may be opened
multiple times in secondary mode. It's not possible to write data to the
database in secondary mode.
Secondary mode does not support checkpoints so there might be data
inconsistencies when data is changed while said data is queried. It
might happen that the result data mixes both elements before and after
the change that can not happen in primary mode.
2 years ago
1fa0633db3
Library: adds basic WASI support
...
Adds tests for WASM+JS and WASI to the CI
2 years ago
854e29ee38
Makes XSD datatypes a standalone crate
...
Allows other crates to reuse Oxigraph XSD datatypes implementation
2 years ago
808c9db007
String formatting: uses inline syntax
2 years ago
59aea75a30
Uses try_into for slice to array conversions
2 years ago
4cea628570
Upgrades to sysinfo 0.26
2 years ago
e9bbe8e036
Applies new Clippy suggestions
2 years ago
95fd439dc2
Updates dependencies
3 years ago
b04bdcceed
Makes the new Clippy happy
3 years ago
ad710dc825
Bulk loader: removes dead code
...
There were some dead code and key-values where deduplicated twice
3 years ago
c815782631
rlimits: be safe if the nice value is bigger than the hard one
3 years ago
cc4e3f8b0f
Bulk loader: avoids to write duplicated values
3 years ago
42f316f7db
Limits the number of opened file by RocksDB
...
This was generating too many open file errors
3 years ago
3cbde014a6
Limits the bulk loader thread count to 4
...
Allows writing bigger files
3 years ago
b5611445eb
Fixes ARM compilation
3 years ago
d0b3d76bf1
Bulk loader: use as much memory as possible
3 years ago
c5f12f10f6
Better RocksDB error Debug implementation
3 years ago
54c6f2c19f
Bulk load: use megabytes as unit
3 years ago
9e969cd8eb
Adds methods to configure bulk loader system consumption
3 years ago
adea1899e5
Bulk loader: rewrite API and add hook to display progress indicators
3 years ago
ac64b58825
Fixes wasm build
3 years ago
822dd60596
Fixes invalid database generated from bulk load
...
Adds an invariant validation test
3 years ago
ad4bfe35c2
Fixes typos in the documentation
3 years ago
203bd4c080
Adds a backup system
...
RocksDB provides a great cheap backup feature thanks to the immutable SST storage.
Sadly it is not compatible with in-memory databases which do not rely on the SST files.
3 years ago
b9ef40df07
Avoids freeing null values
3 years ago
32d3f0782b
Fixes some new clippy warnings
3 years ago
7d58f451bd
Runs LLVM address sanitizer and fixes found bugs
3 years ago
fc26fa87a5
Makes a lot of dumb functions inline
3 years ago
ec5c7c86be
Improves RDF I/O ParseError
3 years ago
f5545f1948
Makes Oxigraph model standalone as oxrdf
...
Allows to reuse it in other crates like spargebra
3 years ago
4337349aad
Moves xsd module to root
...
Makes "model" module more stand-alone
3 years ago
a91ea89fff
Moves str2id outside of transactions
...
Allows avoiding conflicts and reducing transactional guarantees
3 years ago
72a17c4f72
Avoids using std::io::Error where possible
3 years ago
a33dbb6d06
Avoids abusing std::io::Error
...
Introduces new specific errors and make them implement Into<io::Error> for compatibility
3 years ago
18ec80c362
Exposes more of RocksDB internal Status struct
3 years ago
bada850284
Txn conflict: allows the OS to do something else before retrying
3 years ago
44d1a5f04c
Makes bulk load partial insertions atomic
...
Ensures that the stores does not end up in an inconsistent state.
3 years ago
f64001906c
Minor documentation improvements
3 years ago
8d20f65890
Adds back in memory WASM support
3 years ago
Maximilian Goisser