When the frontend makes modification that violate write-permissions or have invlid URIs, the engine now reverts the optimistic changes from the frontend. Additionally, it will notify the frontend so that developers can subscribe to errors that arose.

I created benchmarks for simple shapes and nested shapes (three layers of nesting) and different scenarios. The performance for handling RDF quad updates coming from other sources in nested shape settings was worst so my main focus was on that. The changes benefit initialization speed too though.

For benchmarking, I created a three-layer nested shape. I made a 1000 SPARQL update calls, each adding a new root object with four sub-objects. So eventually there were 5,000 tracked objects. Initially, this took around 6s on my machine. With the improvements it is around 700ms (so a factor of around 8).

Main changes:

• One bottleneck for many ORM functions is the SPARQL engine. Previously, schema_shape_to_sparql would create queries with FILTER, to filter by graphs and subjects. As a consequence, they were not bound. Instead of FILTER, using VALUES improves query speed by a factor of around 4. This benefits updates coming from the database and initialization. On the benchmark, this reduced query-time from around 1,100ms to below 300ms. I suppose future improvements to oxigraph will benefit the overall orm performance in general.
• There were a couple of places in process_changes.rs where I had previously iterated all tracked objects but only needed to find whether new objects already existed (bad O-complexity), see previous use of get_tracked_objects_any_graph and tracked_nested_subjects in init_validation_stack. Improving bad loop complexities reduced the overall time around 2 seconds.
• I removed unnecessary recursion calls for the main process_changes function for validating child objects. Together with that, missing data is now queried earlier and only when necessary. That reduced time by another 1-2 seconds.
• Quad changes coming from the engine are "stopped" earlier by checking if the quads' predicates contain any present in the schema.

Then, there are cases where data needs to be fetched even though the tormos are tracked already (the tormo becomes valid). There was a bug I fixed where the data was not re-fetched. Tests are added.

You can see that I moved a lot of code around. One main change is that a lot of state in the process_changes function is now tracked in a ProcessRun object.

Discussed on the forum (#devs, "NixOS packaging"). Outcome:

• ngd currently needs the two RustEmbed folders to exist…

Two questions, mainly to make sure the packaging tracks the project
correctly rather than guessing:

1. Is there a rough public timeline for an official tagged/self-hostable
   release of the broker? I'm currently building from a pinned main
   commit since there's no release to point nixpkgs at yet — happy to
   keep tracking main if that's the expected path for a while, just
   want to package this in the way that's least disruptive for you
   when a real release does land.
2. Would a NixOS module for ngd (and eventually a nixpkgs PR) be
   something the project wants to be aware of / linked from your docs,
   or is there a preferred packaging convention you'd rather this
   follow?

Context, stated openly: this packaging work is part of a proposal I'm
submitting to NLnet's ELFA fund in November. I know NextGraph is itself
NLnet-funded, so I'd rather you hear that from me than from a reviewer.
The proposal is to package and operate ngd, not to reimplement any of
it — your project is named in it as the upstream this depends on.

Not asking for an endorsement and nothing depends on your answer. I'd
just rather build this in coordination with upstream than in a vacuum.
Happy to share the module and tests if useful.

I keep two commits on top of this for my dev branch: one that setup cargo to use only local checkouts, the other one to disable at rest encryption. Those are not part of this PR and should never land on main.