oxigraph/CHANGELOG.md

[0.3.16] - 2023-04-29

Changed

• Fixes flush and compaction on the GSPO index. It might improve Oxigraph performances and storage space.
• SPARQL: fixes some optimizations in presence quoted triples with nested variables.
• SPARQL profiler: adds EXISTS operation to the explanation and profiling tree.
• Upgrades RocksDB to 8.1.1.

[0.3.15] - 2023-04-18

Added

• Rust: adds Store.explain_query_opt method that allows to get an explanation of the evaluation with profiling statistics.
• Server: adds explanation and profiling to the query action (--explain, --explain-file and --stats options).
• Python: implements the __match_args__ attribute on terms (literals, triples, quads...) to allow match operator usage.
• Server: adds the --cors option to the serve actions to allow cross-origin resource sharing.

Changed

• SPARQL: fixes evaluation of empty aggregation without GROUP BY: aggregators now return their default value (0 for COUNT...) and not an empty row.
• SPARQL: fixes parsing of decimal with more than 19 digits in the fractional part.
• Server docker image: switch to the smaller distroless base images.
• Bulk loader: by default only uses 2 concurrent threads and around 2GB of RAM.
• Server load: progress is now printed to stderr much more regularly.

[0.3.14] - 2023-03-19

Added

• Read only and secondary RocksDB storage. Allows to open the data as read-only or to follow a primary instance.
• Server: adds multiple commands:
  • serve-secondary and serve-read-only to serve the HTTP server in secondary and read-only modes.
  • dump to dump the database content to a file.
  • query and update to execute SPARQL query and updates.
  • backup to do a database backup.
  • optimize to optimize the database storage.
• Server: adds format and graph options to the load command in order to select the loaded data format and the target graph.
• Python: Store.extend and Store.bulk_extend methods.
• Python: allows to pickle basic data model classes.

Changed

• Upgrades RocksDB to 8.0.0.
• Server: listening to localhost now properly listen to both IPv4 and IPv6.
• RDF/XML and XML parser results: avoid an ever growing buffer.
• JSON SPARQL results: allows the "head" key to be at the end of the document.
• TSV SPARQL results: properly quote \t and print trailing line jumps.
• Term::from_str: fixes parsing of blank nodes in object position of quoted triples.
• QuerySolution: implements Eq and Debug.
• JS: Reduces WASM build size.
• OxRDF: fixes Graph and Dataset serialization to output proper N-Triples and N-Quads.

[0.3.13] - 2023-02-23

Changed

• Fixes serialization of times with a decimal number of seconds lower than 10 (now 01:01:01.01 and not 01:01:1.01).
• Turtle and TriG: fixes parsing for prefixes named after keywords (prefix, base, graph, true and false).

[0.3.12] - 2023-02-18

Added

• From<NamedOrBlankNode> for GraphName (and similarly for *Ref).
• Prebuilt Python wheels for Linux with musl libc.

Changed

• Fixes TriG serialization.
• QueryDataset is now properly exposed in the public Rust API.

[0.3.11] - 2023-01-18

Added

• SPARQL: Implementation of the LATERAL operator following SPARQL SEP-0006. Support is behind the sep-0006 feature in spargebra and enabled by default in Oxigraph.
• SPARQL: Implementation of the ADJUST function following SPARQL SEP-0002. Support is behind the sep-0002 feature in spargebra and enabled by default in Oxigraph.
• Rust: There is a new stand-alone crate oxsdatatypes implementing Rust structs for the common XML schema datatypes. It was part of the Oxigraph crate and it might be useful for other SPARQL or XPath implementations.
• Rust: The oxigraph crate can now be compiled for wasm32-wasi with the basic in-memory backend but without RocksDB.

Changed

• SPARQL: The property path evaluator was sometime emitting duplicates when evaluating disjunctive patterns (a|b). It is now fixed.
• SPARQL: If written explicitly in the SPARQL query, the regular expressions are now compiled once and not for each row.
• SPARQL: Property path evaluation with both start and end variables bound has been optimized.
• SPARQL: Casts to xsd:decimal from xsd:float and xsd:double now properly fails on overflow instead of overflowing the internal 128-bits representation. This follows XPath casting rules.
• Rust: The minimal supported Rust version is set at 1.60 and enforced using the CI.
• Python: Local builds will now target the specific Python version instead of abi3. abi3 wheels are still released on Pypi alongside new Python-version specific builds for Linux+GNU.
• SPARQL: Fixes a panic when the estimated upper bound of the results size was overflowing a usize.
• Python: Uses typing.IO in Python stubs instead of narrower interfaces.
• Upgrades RocksDB to 7.9.2, quick-xml to 0.27 and pyo3 to 0.18.

[0.3.10] - 2022-12-21

Added

• SPARQL: Property path with unbound graph are now evaluated. For example, SELECT * WHERE { GRAPH ?g { ?s ex:p+ ?o } } now works.
• SPARQL: The query optimizer is now fuzzed for better testing.

Changed

• SPARQL: Evaluation of zero-length paths with both start and end unbounded now return only terms defined in the current graph but not terms only defined in the query. For example, in SELECT * WHERE { VALUES ?s { 1 } { ?s ex:p? ?o } } the literal 1 won't be returned anymore if not in the queried graph.
• Python: type stubs are now compatible with Python 3.7 and Mypy strict mode.
• RDF/XML: allows entities declared using other entities.
• Upgrades quick-xml to 0.26.

[0.3.9] - 2022-12-07

Added

• Server: The /store endpoints now has a no_transaction HTTP option for POST and PUT request to get better performances at the cost of transactional guarantees.
• Server: The /store endpoints now has a lenient HTTP option for POST and PUT request to ignore syntax errors (requires the no_transaction option).
• Server: allows path that are not valid UTF-8 in file path CLI arguments.
• Rust: From<spargebra::Query> to oxigraph::Query (thanks to @hobofan).

Changed

• SPARQL: NOW() function properly returns the current time and not 1970-01-01
• SPARQL: fixes serialization of SPARQL queries (property path and STRSTARTS function).
• SPARQL: slightly optimize aggregates by avoiding an unneeded projection.
• SPARQL: the parser now cleanly fails if invalid VALUES clauses are provided.
• SPARQL: In DELETE/INSERT UPDATE the currently written values can't be read anymore ("Halloween problem").
• oxrdf: makes Clippy run without warnings when rdf-star is disable.
• Python: makes type annotations compatible with Python 3.7.
• Python: makes sure the parameter default value is always included in the type annotation.

[0.3.8] - 2022-10-22

Changed

• Python: avoid uploading debug wheels to pypi.

[0.3.7] - 2022-10-22

Added

• Python type stubs are now generated and distributed alongside wheels.

Changed

• SPARQL: Fixes evaluation of sub queries in which some variables might not be always bound.
• Python: Uses https://pyoxigraph.readthedocs.io instead of https://oxigraph.org/pyoxigraph/ to store pyoxigraph documentation.
• Server: Migration to Clap v4. Slight changes to the console textual outputs.

[0.3.6] - 2022-08-16

Changed

• Server: Makes sure body is always sent for all HTTP response codes that are not explicitly to not do that according to the HTTP specification.
• Python: Adds support of reading from io.TextIOBase.

[0.3.5] - 2022-08-02

Changed

• Fixes a bug in the SPARQL parser with nested anonymous blank nodes ([]).
• Migrates the Python documentation skin to Furo.

[0.3.4] - 2022-07-17

Changed

• Allows multiple value to be given to oxigraph_server --file parameter.
• Limits parallelism of oxigraph_server file loader to avoid stack overflows.

[0.3.3] - 2022-06-12

Added

• QueryResults::with_http_redirection_limit that allows to set a limit on the number of HTTP redirection followed (0 by default).

Changed

• Fixes a bug in SPARQL parser that was always enabling the SILENT option of the SERVICE operator.
• Allows unknown keys in the objects present in the SPARQL JSON query results. It allows to parse e.g. Virtuoso query results.
• Allows "typed-literal" as an alias of "literal" in SPARQL JSON query results.
• Improves the HTTP client error reporting: no silent failures on 4XX and 5XX responses.
• Upgrades RocksDB to 7.3.1.
• Upgrades quick-xml to 0.23.

[0.3.2] - 2022-04-24

Changed

• Fixes a bug in the server bulk loader that crashed on very small files.
• Upgrades RocksDB to v7.1.
• Removes some dead code in the bulk loader.

[0.3.1] - 2022-04-02

Changed

• The default git branch is now main and not master (thanks to @nyurik).
• Upgrades RocksDB to v7.0.4.
• Limits the number of bulk loader threads to at most 4 (allows bigger BTree chunks and a better data layout).
• Limits the number of files opened by RocksDB to the soft file descriptor limit minus 48.

[0.3.0] - 2022-03-19

Changed

• Fixes compilation on ARM.
• Moves some lints from errors to warnings in order to avoid compilation failures on different Rust versions.

[0.3.0-rc.1] - 2022-03-14

Added

• The bulk loader now checks available memory and tries to increase its batch size to make use of it.
• The Bulk loader provides now a --lenient option to keep loading a file even if invalid data is found (works only with N-Triples and N-Quads). This behavior can be customised in the Rust API using the BulkLoader::on_parse_error method.

Changed

• Rocksdb has been upgrade to 7.0.2. It now requires a C++17 compatible compiler. This required dropping support of macOS 10.9 to 10.13.

[0.3.0-beta.4] - 2022-02-27

Added

• JS: Oxigraph NPM package is now also supporting web browsers and WebPack.
• JS: RDF term related classes now overrides the toString method.
• Python: It is now possible to directly give a file path to the parse, serialize, Store.load, Store.bulk_load and Store.dump functions.
• Python: New Store.clear_graph, Store.clear, Store.optimize and Store.flush methods.

Removed

• sophia_api traits implementation following a request of Sophia maintainer.

Changed

• SPARQL: fixes evaluation of SPARQL queries with no results but an ORDER BY clause. There should be no group in the output instead of one empty group. This behavior has been changed following this discussion.
• SPARQL: fixes SPARQL-star evaluation of nested triples with both variables and constants.
• SPARQL: if results are sorted, literals are now ordered by value, then datatype, then language tag. This ordering is considered as "implementation defined" by the SPARQL specification and is very likely to change in the future.
• Python: all costly methods now release the python GIL allowing multithreaded usages of pyoxigraph.
• Rust: SPARQL results writer now flushes the buffer at the end of the results writes. This makes their API less error-prone.
• Rust: the bulk loader API has been rewritten to allow hooking a progress indicator and set parallelism limit.
• Server: it is now possible to bulk load gzipped files.

[0.3.0-beta.3] - 2022-02-02

Changed

• Fixes a bug in the bulk_load_dataset method that was creating an invalid database.
• Server: Takes into account also URL query parameters if the send SPARQL request body is using form-urlencoded.
• Upgrades RocksDB to v0.28.2.
• Generate clean python sdist files compiling Oxigraph from scratch with the proper Cargo.lock.
• Do not push beta releases to homebrew and python stable documentation.
• Moves RocksDB binding directory to oxrocksdb-sys.

[0.3.0-beta.2] - 2022-01-29

Changed

• Fixes release on crates.io of the RocksDB bindings.

[0.3.0-beta.1] - 2022-01-29

Added

• RDF-star support. Triple is now a possible Term. Serialization formats and SPARQL support have been updated to match the latest version of the specification draft.
• Fast data bulk load with the Store bulk_load_dataset and bulk_load_graph methods and a special command line option of the server.
• It is now possible to quickly backup the database using the backup method.
• Rust: *Syntax::from_extension to easy guess a graph/dataset/sparql result format from a file extension.
• Rust: Custom SPARQL functions are now supported using QueryOptions::with_custom_function.
• Rust: Simple in-memory graph (Graph) and dataset (Dataset) data structures with canonicalization.
• Nightly build of the server binary and docker image, and of pyoxigraph wheels.
• Store operations are now transactional using the "repeatable read" isolation level: the store only exposes changes that have been "committed" (i.e. no partial writes) and the exposed state does not change for the complete duration of a read operation (e.g. a SPARQL query) or a read/write operation (e.g. a SPARQL update). the Store transaction method now allows to do read/write transactions. -RDF-star <https://w3c.github.io/rdf-star/cg-spec>_ is now supported (including serialization formats and SPARQL-star). :py:class:.Triple can now be used in :py:attr:.Triple.object, :py:attr:.Triple.object, :py:attr:.Quad.subject and :py:attr:.Quad.object.

Changed

• SPARQL: It is now possible to compare rdf:langString literals with the same language tag.
• SPARQL: The parser now validates more carefully the inputs following the SPARQL specification and test suite.
• SPARQL: Variable scoping was buggy with "FILTER EXISTS". It is now fixed.
• Rust: RDF model, SPARQL parser and SPARQL result parsers have been moved to stand-alone reusable libraries.
• Rust: HTTPS is not supported by default with the http_client option. You need to enable the native-tls or the rustls feature of the oxhttp crate to enable a TSL layer.
• Rust: The error types have been cleaned. Most of the Store methods now return a StorageError that is more descriptive than the previous std::io::Error. The new error type all implements Into<std::io::Error> for easy conversion.
• Rust: There is now a Subject struct that is the union of NamedNode, BlankNode and Triple. It is The used type of the subject field of the Triple and Quad structs.
• Rust: The SPARQL algebra is not anymore publicly exposed in the oxigraph crate. The new oxalgebra crate exposes it.
• Rust: UpdateOptions API have been rewritten. It can now be built using From<QueryOptions> or Default.
• Server: The command line API has been redesign. See the server README for more information.
• Server: The HTTP implementation is now provided by oxhttp.
• Server: The HTTP response bodies are now generated on the fly instead of being buffered.
• Python: The SledStore and MemoryStore classes have been removed in favor of the Store class.
• JS: The MemoryStore class has been renamed to Store.
• JS: The RDF/JS DataFactory interface is now implemented by the oxigraph module itself and the MemoryStore.dataFactory property has been removed.
• The implementation of SPARQL evaluation has been improved for better performances (especially joins).
• The TLS implementation used in SPARQL HTTP calls is now rustls and not native-tls. The host system certificate registry is still used.
• Spargebra: The basic RDF terms are now the ones of the oxrdf crate.

Removed

• SledStore and MemoryStore. There is only the RocksDbStore anymore that is renamed to Store.
• oxigraph_wikibase is now stored in its own repository.
• Rust: From implementations between oxigraph terms and rio_api terms.

Many thanks to Thad Guidry, James Overton and Jeremiah who sponsored the project during the development of this version.

[0.2.5] - 2021-07-11

Added

• SPARQL 1.1 Query Results JSON Format parser.
• Python wheels for macOS are now universal2 binaries.

Changed

• The Cargo.lock file is now provided with releases to avoid compilation failures because of changes in dependencies.
• Uses clap instead of argh for the server arguments parsing.
• Upgrades PyO3 to v0.14.

[0.2.4] - 2021-04-28

Changed

• The HTTP server allows to query the union of all graphs using the union-default-graph query parameter and to use the union graph for update WHERE clauses using the using-union-graph parameter.
• Exposes Sled flush operation (useful for platforms without auto-flush like Windows or Android).
• Fixes a possible out of bound panic in SPARQL query evaluation.
• Upgrades RocksDB to 6.17.3.

[0.2.3] - 2021-04-11

Changed

• Server: Fixes HTTP content negotiation (charset constraints, failure to properly handle */*...).
• Makes Clippy 1.51 happy.

[0.2.2] - 2021-03-18

Added

• Support of XML entities to the RDF/XML parser

Changed

• Serve: Allows unsupported query parameters in HTTP SPARQL requests.
• Fixes WASM compilation bug and optimises WASM release packages.
• Fixes named graph creation inside of a SledStore transaction.

[0.2.1] - 2021-01-16

Changed

• Fixes pyoxigraph build by enforcing a given maturin version.
• Adds code to build Python wheels for MacOS and Windows.

[0.2.0] - 2021-01-07

Added

• SPARQL 1.1 Update support for Rust, Python and JavaScript. All store-like classes now provide an update method.
• SPARQL 1.1 Query Results CSV and TSV Formats serializers and TSV format parser.
• SPARQL 1.1 Graph Store HTTP Protocol partial support in oxigraph_server. This protocol is accessible under the /store path.
• The SPARQL Query and Update algebra is now public.
• The stores are now "graph aware" i.e. it is possible to create and keep empty named graphs.
• A simple built-in HTTP client. In the Rust library, is disabled by default behind the http_client feature. It powers SPARQL federation and SPARQL UPDATE LOAD operations.
• std::str::FromStr implementations to NamedNode, BlankNode, Literal, Term and Variable allowing to easily parse Turtle/SPARQL serialization of these terms.
• Optional Sled storage for oxigraph_server.

Removed

• The default_graph_uris and named_graph_uris parameters from pyoxigraph query methods.
• Python 3.5 support.
• (Memory|RocksDB|Sled)Store::prepare_query methods. It is possible to cache SPARQL query parsing using the Query::parse function and give the parsed query to the query method.

Changed

• Loading data into oxigraph_server is now possible using /store and not anymore using /. For example, you should use now curl -f -X POST -H 'Content-Type:application/n-quads' --data-binary "@MY_FILE.nq" http://localhost:7878/store to add the N-Quads file MY_FILE.nt to the server dataset.
• Fixes evaluation of MONTH() and DAY() functions on the xsd:date values.
• Variable::new now validates the variable name.
• (Memory|RocksDB|Sled)Store::query does not have an option parameter anymore. There is now a new query_opt method that allows giving options.
• xsd:boolean SPARQL function now properly follows XPath specification.
• Fixes SPARQL DESCRIBE evaluation.

Disk data format

The disk data format has been changed between Oxigraph 0.1 (version 0) and Oxigraph 0.2 (version 1). Data is automatically migrated from the version 0 format to the version 1 format when opened with Oxigraph 0.2.

[0.1.1] - 2020-08-14

Added

• The "sophia" feature implementing the sophia_api traits on Oxigraph terms and stores.
• Explicit types for quads iterators returned by stores.

Changed

• QueryOptions::with_default_graph now takes an impl Into<GraphName> instead of an impl Into<NamedNode>.
• QueryOptions::with_named_graph now takes an impl Into<NamedOrBlankNode> instead of an impl Into<NamedNode>.
• pyoxigraph query methods now takes two new parameters, default_graph and named_graphs. default_graph_uris and named_graph_uris parameters are deprecated.
• Fixes a bug in xsd:gYear parsing.

[0.1.0] - 2020-08-09

Added

• QueryOptions now allows settings the query dataset graph URIs (the SPARQL protocol default-graph-uri and named-graph-uri parameters).
• pyoxigraph store query methods allows to provide the dataset graph URIs. It also provides an option to use all graph names as the default graph.
• "default graph as union option" now works with FROM NAMED.
• pyoxigraph now exposes and documents Variable, QuerySolution, QuerySolutions and QueryTriples

[0.1.0-rc.1] - 2020-08-08

Added

• oxigraph Rust library with SPARQL 1.1 query support and memory, Sled and RocksDB stores.
• oxigraph_server standalone SPARQL server.
• oxigraph_wikibase standalone SPARQL server loading data from a Wikibase instance.
• pyoxigraph Python library based on Oxigraph.
• oxigraph NodeJS library based on Oxigraph.