@ -4,6 +4,6 @@ description: Interact with ActivityPub social networks from NextGraph, thanks to
layout: ../../layouts/MainLayout.astro
---
Thanks to our collaboration with the [ActivityPods project](https://activitypods.org) we will soon offer full compatibility with [ActivityPub](https://activitypub.rocks/) network. At the same time, all apps developed on ActivityPods framework will be compatible with NextGraph.
Thanks to our collaboration with the [ActivityPods project](https://activitypods.org/activitypods-and-nextgraph-are-joining-forces) we will soon offer full compatibility with [ActivityPub](https://activitypub.rocks/) network. At the same time, all apps developed on ActivityPods framework will be compatible with NextGraph.
- they participate between each other, to the Core Network, which is a P2P network with topology of a **general undirected graph**, using the Core Protocol.
- they expose an interface to Verifiers that use the Client Protocol. this is a typical client-server communication, with star topology.
- they expose an interface to Verifiers that use the Client Protocol. This is a typical client-server communication, with star topology.
A verifier only connects to one broker at a time, and needs to register and authenticate in order to access the broker.
Verifiers do not talk to each other (Except in some rare cases when reconciliation is needed. not implemented yet). Instead they talk to their Broker, and it is the Broker who does all the job of maintaining the pub/sub and forwarding events.
Verifiers do not talk to each other (Except in some rare cases when reconciliation is needed. Not implemented yet). Instead they talk to their Broker, and it is the Broker who does all the job of maintaining the pub/sub and forwarding events.
Eventually, an Application connects to a Verifier (locally or remotely) and exchanges with it using the App protocol. This protocol exchanges plaintext data.
@ -18,7 +18,7 @@ Each Document has a double nature :
It has a **_Document-like_** nature where you can store and edit some rich-text, plaintext, or just some JSON or XML data, according to your needs and the primary class of the Document that you have chosen when you created it. This is so far very consistent with what we expect from a document in general. All the apps that you use or develop with NextGraph, will store their data inside these Documents. The Document-like nature is represented with a "cloud" icon. A note for developers: In the internals of NextGraph, the Document-like nature of a Document is called "discrete", because otherwise it is too many things that are called "document".
A Document also has a **_Graph_** nature, which is something new that NextGraph added (hence the name "NextGraph"). This "graph nature" of the document let's you link this document to other documents. It also let's you enter some specific data or meta-data about this document, that will be part of the Graph of all your documents. This graph is something important, that you are not used to. Social networks are all based on Graphs. The Web itself is a huge Graph. When you follow or get followed, when you like or comment on a Post, when you write a DM to someone, all this information is stored as a Graph, that connects different documents together. Later, you will probably need to _query_ this graph. This is done transparently when you want to see all your followers, and when you want to consult the **stream** of all the posts that they have published recently, or when you want to search for something. It is also used for recommendations. But in any case, what is important to understand is that internally, each Document can be linked to any other Document (like the classical Web that links webpages) and that the applications you will use on NextGraph, will also store more Graph information. Because NextGraph is local-first and decentralized, this **graph** information is available to you at all time. And if you want to see it, you can go to any Document and in the Document Menu, you can select "Graph" and you will see options to view and edit the Graph. This is a bit technical for now (you will see things like Turtle, SPARQL etc) but in the future, we will provide here some nice tools where you will be able to explore your own graph easily. The Graph-like nature is represented with a "sun" icon. The Graph part of the document is stored in the RDF format.
A Document also has a **_Graph_** nature, which is something new that NextGraph added (hence the name "NextGraph"). This "graph nature" of the document lets you link this document to other documents. It also lets you enter some specific data or meta-data about this document, that will be part of the Graph of all your documents. This graph is something important, that you are not used to. Social networks are all based on Graphs. The Web itself is a huge Graph. When you follow or get followed, when you like or comment on a Post, when you write a DM to someone, all this information is stored as a Graph, that connects different documents together. Later, you will probably need to _query_ this graph. This is done transparently when you want to see all your followers, and when you want to consult the **stream** of all the posts that they have published recently, or when you want to search for something. It is also used for recommendations. But in any case, what is important to understand is that internally, each Document can be linked to any other Document (like the classical Web that links webpages) and that the applications you will use on NextGraph, will also store more Graph information. Because NextGraph is local-first and decentralized, this **graph** information is available to you at all time. And if you want to see it, you can go to any Document and in the Document Menu, you can select "Graph" and you will see options to view and edit the Graph. This is a bit technical for now (you will see things like Turtle, SPARQL etc) but in the future, we will provide here some nice tools where you will be able to explore your own graph easily. The Graph-like nature is represented with a "sun" icon. The Graph part of the document is stored in the RDF format.
In addition, you can also attach some **binary files** to any Document. Those files are immutable in the sense that you can add them or delete them, but you cannot modify their content.
@ -26,7 +26,7 @@ So to recap, in a Document we have :
- the **Graph** part (some RDF triples). This is mandatory and always available in all documents, even if left empty.
- the **Discrete** part (some JSON, XML or plaintext. based on Yjs or Automerge CRDTs). This can be optional, meaning that it is possible to create a purely graph-based document that doesn't hold any discrete part.
- the **Discrete** part (some JSON, XML or plaintext. Based on Yjs or Automerge CRDTs). This can be optional, meaning that it is possible to create a purely graph-based document that doesn't hold any discrete part.
- some optional **binary files** attached to the document.
@ -48,7 +48,7 @@ It follows the **DID** (Decentralized Identifier) format, and an adhoc DID Metho
When you create a new Document, you first have to select its **type**, which is also called "primary class" internally.
There are already several types proposed to you (not all of them are currently implemented. see a list in [features](/en/features) ), and App developers can add new types.
There are already several types proposed to you (not all of them are currently implemented. See a list in [features](/en/features) ), and App developers can add new types.
Here is a short list of the most used types :
@ -108,9 +108,9 @@ To be more precise, this primary class and all those commits, are not stored for
A **branch** is a concept that helps us separate the data within a single document, and it can be used in several ways.
- A document can be composed of several blocks. Imagine the blocks like separate paragraphs in a text document, or separate embeds (video, table, spreadsheet, etc...) of content that come from other documents or that have different types, and you want to assemble all those blocks into one document. So you select those blocks and put them one after another. When the reader opens a document, they will not see that it is made of several blocks. to them it will just appear to be one document that they scroll into.
- A document can be composed of several blocks. Imagine the blocks like separate paragraphs in a text document, or separate embeds (video, table, spreadsheet, etc...) of content that come from other documents or that have different types, and you want to assemble all those blocks into one document. So you select those blocks and put them one after another. When the reader opens a document, they will not see that it is made of several blocks. To them it will just appear to be one document that they scroll into.
- We also use branches when we want to fork the work that is happening on a Document (let's say in collaborative editing). This can be handy if we want to keep the current version of the Document unchanged, while we are working on a newer version. All the work on the newer version is not visible to the reader, until the "working branch" is merged back into the main branch. This is something that programmers will understand very well, but that can be very handy for everybody else too. Please note that we use the terms "fork" and "merge" here as we did when we talked about CRDTs earlier, but it is not exactly the same thing. Here the fork and merge happens only on request from the user. You can decide at some point to fork a document or block. Then you can also decide that it is time to merge it back. But that's a bit different than the automatic and transparent "merge and fork" that is continuously happening every time that someone goes offline. To make things more clear, we don't call the later this way, we just say the document synchronizes itself. fork and merge should be terms used only when the user decides to manually fork.
- We also use branches when we want to fork the work that is happening on a Document (let's say in collaborative editing). This can be handy if we want to keep the current version of the Document unchanged, while we are working on a newer version. All the work on the newer version is not visible to the reader, until the "working branch" is merged back into the main branch. This is something that programmers will understand very well, but that can be very handy for everybody else too. Please note that we use the terms "fork" and "merge" here as we did when we talked about CRDTs earlier, but it is not exactly the same thing. Here the fork and merge happens only on request from the user. You can decide at some point to fork a document or block. Then you can also decide that it is time to merge it back. But that's a bit different than the automatic and transparent "merge and fork" that is continuously happening every time that someone goes offline. To make things more clear, we don't call the later this way, we just say the document synchronizes itself. Fork and merge should be terms used only when the user decides to manually fork.
- Branches and blocks can also be used to store different translation of your content. You do not want to create a new document for each translation. Instead, you create separate blocks within the same document, and each block holds the translated content. NextGraph is designed to help you with managing translation and multi-lingual content, and when the user will open the Document, the most adapted language will be selected for them.
@ -118,13 +118,13 @@ A **branch** is a concept that helps us separate the data within a single docume
- Also, we internally use the branches feature to store some internal data that you don't see, but that needs to have different branches for the good functioning of NextGraph.
One of the advantage of dividing your content into several blocks is also that blocks can have different read permissions. So if you want to share some information with a certain group of people, but not with some other group, but still refer to that as one document, then you can use blocks. By example, let's say you have a Social profile, where you describe yourself, your interest, and where you put some contact information. Maybe the contact details should not be available to everyone (like phone number, postal address, etc..) Still, you do not want to create 2 documents for that purpose. So you will have 2 blocks. one will only contain the minimum public information you want to share to everyone. The other block will contain more details like phone number and address, and it will include also a link to the other block. This way, you don't have to copy paste the general information about you in both blocks. The "include link" is transparent for the privileged reader who got access to the more private profile. they will not see that the document is composed of 2 blocks. to them it will just appear as one document. And of course, all the other readers who only got the link to the generic block, will not see that there exists another block that contains more private details.
One of the advantage of dividing your content into several blocks is also that blocks can have different read permissions. So if you want to share some information with a certain group of people, but not with some other group, but still refer to that as one document, then you can use blocks. By example, let's say you have a Social profile, where you describe yourself, your interest, and where you put some contact information. Maybe the contact details should not be available to everyone (like phone number, postal address, etc..) Still, you do not want to create 2 documents for that purpose. So you will have 2 blocks. One will only contain the minimum public information you want to share to everyone. The other block will contain more details like phone number and address, and it will include also a link to the other block. This way, you don't have to copy paste the general information about you in both blocks. The "include link" is transparent for the privileged reader who got access to the more private profile. They will not see that the document is composed of 2 blocks. To them it will just appear as one document. And of course, all the other readers who only got the link to the generic block, will not see that there exists another block that contains more private details.
So, to recap. Each branch holds a separate list of commits, that can be seen in the History pane.
For now the App does not let you create new blocks and branches, but it will come very soon. internally, the branches are already there, and they work well. What you see for now in the App when you open a document, is called the "main" branch. Later, you will be able to add more blocks and branches.
For now the App does not let you create new blocks and branches, but it will come very soon. Internally, the branches are already there, and they work well. What you see for now in the App when you open a document, is called the "main" branch. Later, you will be able to add more blocks and branches.
Those who are more curious about the internals of the branch, commits, DAG (directly acyclic graph) used in NextGraph, can refer to the [Repo format](/en/specs/format-repo) of the specs.
Those who are more curious about the internals of the branch, commits, DAG (Directed Acyclic Graph) used in NextGraph, can refer to the [Repo format](/en/specs/format-repo) of the specs.
### Stores
@ -142,7 +142,7 @@ Those are :
- the **protected** store : is a space where you can share data, documents, and media with other users, but they will need a special link and permission in order to access them and collaborate with you on those documents (or just read them). You can also create Groups here when you want to chat or collaborate with several users. This store also acts as your protected profile for social networks. Only your followers will be able to see this profile, as it is not public.
- the **private** store : this is a place where you put only private and personal information that only you have access to. This store will be accessible to all your devices, so it is useful in order to synchronize private data between them. You can put sensitive data too, as everything is encrypted. Nobody else will ever see this data. it is not possible to share the documents of your private store with anybody else.
- the **private** store : this is a place where you put only private and personal information that only you have access to. This store will be accessible to all your devices, so it is useful in order to synchronize private data between them. You can put sensitive data too, as everything is encrypted. Nobody else will ever see this data. It is not possible to share the documents of your private store with anybody else.
You can also later, move documents from one store to another, if you are the owner of such document.
@ -154,7 +154,7 @@ And your public profile/store can be found with the icon that represents one use
We said that you can create **Groups** when you want to share, exchange and collaborate with other users. In fact, each Group is a separate Store. This is helpful because each group can have its own set of permissions, and then you can configure the store so that all the documents included in this store, inherit the permissions of the store. This way, we can manage the group easily. If we add a member to the group, they immediately get access to all the documents in the group. The same happens when you remove a user from the group: they loose access to all the documents at once.
Because most of the time, collaboration on Documents happens within a group of users, that will probably share more than one document between each other, the Store regroups all the Documents that such Group wants to interact with. Not to confuse this Group with the E2EE group i was referring to in the [Encryption chapter](/en/encryption). In fact, in NextGraph terminology, we never talk about any E2EE group. Instead we call that a Repo, or a Document. A Repo is basically the equivalent of an E2EE group for one and only one Document. So when we talk about a Group, we are referring instead to a Store that gathers all the users and their peers, and where they will be able to share several Documents (or Repos if you prefer).
Because most of the time, collaboration on Documents happens within a group of users that will probably share more than one document between each other, the Store regroups all the Documents that such Group wants to interact with. Not to confuse this Group with the E2EE group i was referring to in the [Encryption chapter](/en/encryption). In fact, in NextGraph terminology, we never talk about any E2EE group. Instead we call that a Repo, or a Document. A Repo is basically the equivalent of an E2EE group for one and only one Document. So when we talk about a Group, we are referring instead to a Store that gathers all the users and their peers, and where they will be able to share several Documents (or Repos if you prefer).
As you will see, a Store can be organized into folders and sub-folders, making it very easy to keep tidy. It is the equivalent of the "drive" that you have been using in cloud-based sharing system.
@ -26,7 +26,7 @@ Here is the breakdown of the problem.
DNS and TLS certificates are not fully decentralized and they both rely on some single points of failure (and domains are expensive, and hard to setup and manage for the end-user). TLS was discarded by the creator of Signal Protocol by example, because his main goal was to offer a secure and totally decentralized chat protocol. We also have to forgo DNS and TLS in our architecture, for similar reasons. That's not an easy task. Matrix, by example, that is just another iteration on the above mentioned Signal Protocol (reimplemented in permissive licensing, and with a slightly different protocol for group encryption), felt the need to wrap all the E2EE protocol within some classical REST apis with TLS and a domain name, so basically they voided the advantage that Signal had created: no need for DNS nor TLS certificates. The result is Synapse protocol and Home Server: a mix of E2EE and plain old TLS APIs. That's not what we did. In our protocol, every peer is known by its IP and a PeerID that is a public key. There is no DNS, there is no TLS certificate. We use the Noise protocol for encryption of communication between peers, which is, for now, channeled into a WebSocket, but could be transported over other protocols, including UDP, QUIC or WebRTC, by example.
### P2P isn't viable. we need 2-Tier network with Brokers
### P2P isn't viable. We need 2-Tier network with Brokers
Pure Peer to Peer (P2P) is not practical in real life. Even though we claim NextGraph is peer-to-peer (and it is, somehow), our network topology is not exactly peer-to-peer. What we do is called “**2-tier topology**". It means that the peers/replicas do not communicate directly with each other. Instead, they send and receive their data from some tiers that we call “**Brokers**”. The broker is a server that is always up and running, and that replicas can always contact in order to sync their data. To the contrary, a truly P2P topology would imply that the replicas are connecting directly to each other. This is what HyperHyperSpace is doing by example, with WebRTC that establishes connections directly between web-browsers. Several other protocols are doing direct P2P communication. While it looks good on the paper, and is attractive because it seems at first glance as a desirable feature, it turns out that true P2P is not practical because it forces the peers/replicas that want to sync and exchange data, to be both online at the very same moment. This condition is not guaranteed at all and will impede synchronization if not met. What are the chances that any participant of a collaborative group are connected to the internet at the very same time? very low in real life cases. If I want to sync between several of my own devices, does that mean that at least 2 of my devices have to be online at the same time? this is not gonna work. For this reason, we opted for a “2-tier” topology. Peers that are intermittently connected to the internet, can still synchronize because they will talk only to a common “broker” that is always up and running and that is there specially for that. In fact each broker is connected to the other brokers in the overlay, so it doesn't matter to which broker the client is connected.
@ -42,9 +42,9 @@ Like with the double ratchet, we want to renew epoch from time to time. Speciall
### A repo has editors, viewers and signers
In NextGraph, **each Document** has its own E2EE group, that we call a **Repo**. Now, a Repo has editors, obviously, and those should be inside the E2EE group. Sure thing. Then, we also have users that are only “readers”. they cannot edit the document. Should they be part of the E2EE group? how are the permissions going to be enforced? who is going to check if this specific user can edit or not, in a local-first setting? NextGraph uses the concept of capabilities (OCAP) for permissions. A user can read the content of a document if they have the capability to do so. Furthermore, the capability system that we have designed, is not a traditional OCAP mechanism where a “controller” eventually decides on what to do with an incoming action and associated capability. Instead, in NextGraph, there is no need for an almighty controller. We have designed a capability mechanism based on cryptography. If the user possesses the decryption key, then they are entitled to read the data. There is no need for a “controller” to verify the signature of the capability/delegation they hold. Instead, the user just gets some encrypted data, and if they have the correct decryption key, then it means they will be able to decrypt and read the data. if they don't, well, they will see garbage instead. About the “write” permission, it is the same. The editor needs to possess the private key of the Pub/Sub topic that is used to propagate the edits (more on the pub/sub later). So basically, if a user wants to modify the data, they can try as much as they want locally, but if they don’t have the private key of the pub/sub topic, it will not work because they will not be able to propagate their updates in the overlay (the brokers will refuse it). Again, there is no need for a “controller” here that will check a capability, or better said: the capability is included in the message. The “write” permission is just enforced with cryptography.
In NextGraph, **each Document** has its own E2EE group, that we call a **Repo**. Now, a Repo has editors, obviously, and those should be inside the E2EE group. Sure thing. Then, we also have users that are only “readers”. They cannot edit the document. Should they be part of the E2EE group? how are the permissions going to be enforced? who is going to check if this specific user can edit or not, in a local-first setting? NextGraph uses the concept of capabilities (OCAP) for permissions. A user can read the content of a document if they have the capability to do so. Furthermore, the capability system that we have designed, is not a traditional OCAP mechanism where a “controller” eventually decides on what to do with an incoming action and associated capability. Instead, in NextGraph, there is no need for an almighty controller. We have designed a capability mechanism based on cryptography. If the user possesses the decryption key, then they are entitled to read the data. There is no need for a “controller” to verify the signature of the capability/delegation they hold. Instead, the user just gets some encrypted data, and if they have the correct decryption key, then it means they will be able to decrypt and read the data. If they don't, well, they will see garbage instead. About the “write” permission, it is the same. The editor needs to possess the private key of the Pub/Sub topic that is used to propagate the edits (more on the pub/sub later). So basically, if a user wants to modify the data, they can try as much as they want locally, but if they don’t have the private key of the pub/sub topic, it will not work because they will not be able to propagate their updates in the overlay (the brokers will refuse it). Again, there is no need for a “controller” here that will check a capability, or better said: the capability is included in the message. The “write” permission is just enforced with cryptography.
So, coming back to the problem of the E2EE group, we have several types of users with different types of access, that interact within the Repo. in addition to the **editors** and the **readers**, we also have **signers**. Those signers have only one task: Verify and Sign the commits, on request. They are not necessarily editors. But they do need read access in order to read what they are going to sign. So we have **3 types of access** (and a 4th type for the owners of the documents). Anyway. it starts to seem obvious that a classical E2EE group like OpenMLS or Matrix will not be enough, because we do not want the readers, by example, to see what the editors are doing. We don't even want them to see the list of editors. Are we going to create 4 different OpenMLS group? one for each access type? No. That would not work in practice, and not even on the paper.
So, coming back to the problem of the E2EE group, we have several types of users with different types of access, that interact within the Repo. In addition to the **editors** and the **readers**, we also have **signers**. Those signers have only one task: Verify and Sign the commits, on request. They are not necessarily editors. But they do need read access in order to read what they are going to sign. So we have **3 types of access** (and a 4th type for the owners of the documents). Anyway, it starts to seem obvious that a classical E2EE group like OpenMLS or Matrix will not be enough, because we do not want the readers, by example, to see what the editors are doing. We don't even want them to see the list of editors. Are we going to create 4 different OpenMLS group? one for each access type? No. That would not work in practice, and not even on the paper.
So here we are, we designed and implemented a new E2EE group protocol, that takes into account the specific needs of Document editing, signing and sharing, together with cryptographic capabilities. And this… did not exist, as far as we know. Add to this the fact that we also do content-addressing and deduplication with convergent encryption on all the content and pub/sub messages, and you will soon understand why a novel protocol was needed.
Note: (\*) The JSON Editor (in read-write or read-only) is very primitive for now, it has been developed in one day, to showcase the easy binding of JSON data to some Svelte components. It lacks important features like ( insert in array at position. remove at position in array. splice range. remove from map. change type or name of property in map ). Also the general UX is ugly. But it is functional and demonstrate the potential of CRDT on JSON data, that will be leveraged by app developers. Once it will be improved, this Editor and Viewer will be reused for all Data types, including the Yjs based ones, and the Graph one. It also showcases that no matter what the App is doing, you own the data and can always go to see it and even modify it, or export it. That's what a data-centric framework is about.
Note: (\*) The JSON Editor (in read-write or read-only) is very primitive for now, it has been developed in one day, to showcase the easy binding of JSON data to some Svelte components. It lacks important features like ( insert in array at position, remove at position in array, splice range, remove from map, change type or name of property in map ). Also the general UX is ugly. But it is functional and demonstrate the potential of CRDT on JSON data, that will be leveraged by app developers. Once it will be improved, this Editor and Viewer will be reused for all Data types, including the Yjs based ones, and the Graph one. It also showcases that no matter what the App is doing, you own the data and can always go to see it and even modify it, or export it. That's what a data-centric framework is about.
Note: (\*\*) The JSON Editor (in read-write or read-only) used here for the Yjs types, is based on svelte-jsoneditor which isn't so easy to use. But it is a good demonstration that the GUIs can be anything. The data layer works fine and that's the most important for now.
description: NextGraph supports several CRDTs natively, and is open to integrating more of them in the future. more about the Unified data model and features of our CRDTs
description: NextGraph supports several CRDTs natively, and is open to integrating more of them in the future. More about the Unified data model and features of our CRDTs
layout: ../../../layouts/MainLayout.astro
---
@ -14,7 +14,7 @@ NextGraph supports several CRDTs natively, and is open to integrating more of th
For now, we offer:
- **Graph** CRDT : the Semantic Web / Linked Data / RDF format, made available as a local-first CRDT model thanks to an OR-set logic (Observe Remove Set) formalized in the paper [SU-set (SPARQL Update set)](https://inria.hal.science/hal-00686484/document). This allows the programmer to link data across documents, globally, and privately. Any NextGraph document features a Graph, that enables connecting it with other documents, and representing its data following any Ontology/vocabulary of the RDF world. Links in RDF are known as predicates and help establishing relationships between Documents while qualifying such relation. Learn more about RDF and how it allows each individual key/value to point to another Document/Resource, similar to foreign keys in the SQL world. With the SPARQL language you can then traverse the Graph and navigate between Documents.
- **Graph** CRDT : the Semantic Web / Linked Data / RDF format, made available as a local-first CRDT model thanks to an OR-set logic (Observed Remove Set) formalized in the paper [SU-set (SPARQL Update set)](https://inria.hal.science/hal-00686484/document). This allows the programmer to link data across documents, globally, and privately. Any NextGraph document features a Graph, that enables connecting it with other documents, and representing its data following any Ontology/vocabulary of the RDF world. Links in RDF are known as predicates and help establishing relationships between Documents while qualifying such relation. Learn more below about RDF and how it allows each individual key/value to point to another Document/Resource, similar to foreign keys in the SQL world. With the SPARQL language you can then traverse the Graph and navigate between Documents.
- **Automerge** CRDT: A versatile and compact format that offers sequence and set operations in an integrated way, that lets all types be combined in one document. It is based on the formal research of [Martin Kleppmann](https://martin.kleppmann.com/2020/07/06/crdt-hard-parts-hydra.html), Geoffrey Litt et al. and the Ink & Switch lab, implemented by Alex Good, and follows the [RGA algorithm](https://liangrunda.com/posts/automerge-internal-1/) ([Replicated Growable Array](https://www.sciencedirect.com/science/article/abs/pii/S0743731510002716)). Their work brought the formalization of Byzantine Eventual Consistency, upon which NextGraph builds its own design. Automerge offers a rich text API (Peritext) but we do not expose it in NextGraph for now, preferring the one of Yjs for all rich text purposes.
@ -22,7 +22,7 @@ For now, we offer:
In the future, we might want to integrate more CRDTs into NextGraph, specially the promising LORO or json-joy, or older CRDTs like Diamond Types.
If you are not familiar with CRDTs, here are some nice introductions you can read about that subject [here](https://www.farley.ai/posts/causal) and [here](https://www.bartoszsypytkowski.com/the-state-of-a-state-based-crdts/).
If you are not familiar with CRDTs, there are some nice introductions you can read about that subject [here](https://www.farley.ai/posts/causal) and [here](https://www.bartoszsypytkowski.com/the-state-of-a-state-based-crdts/).
## Semantic Web, Linked Data, RDF
@ -64,62 +64,62 @@ As you will see in the **Sync Protocol** chapter below, the programmer decides w
Now let's have a look at what those CRDTs have in common and what is different between them. We have marked 🔥 the features that are unique to each model and that we find very cool.
| <tdcolspan=3> (aka property/value, and in RDF, it is called predicate/object) this is the basic feature that the 3 models offer. You have a Document, and you can add key/value pairs to it. Also known as Map or Object. |
| <tdcolspan=3> Thanks to the Ontology/Schema mechanism of RDF (OWL), the schema information is embedded inside the data (with what we call a Predicate), thus avoiding any need for schema migration. Also, it brings full data interoperability as many common ontologies already exist and can be reused, and new ones can be published or shared |
| **nested** | ✅ blank nodes | ✅ | ✅ |
| <tdcolspan=3> key/value pairs can be nested (like in JSON) |
| **sequence** | ❌ \* | ✅ | ✅ |
| <tdcolspan=3> And like in JSON or Javascript, some keys can be Arrays (aka list), which preserve the ordering of the elements. (\*) In RDF, storing arrays is a bit more tricky. For that purpose, Collections can encode order, but are not CRDT based nor concurrency safe |
| **sets** | 🔺 multiset | ✅ | ✅ |
| <tdcolspan=3> RDF predicates (the equivalent of properties or keys in discrete documents) are not unique. they can have multiple values. that's the main difference between the discrete and graph models. We will offer the option to enforce rules on RDF data (with SHACL/SHEX) that could force unicity of keys, but that would require the use of **Synchronous Transactions**. Sets are usually represented in JS/JSON with a map, of which the values are not used (set to null or false), because keys are unique, so we use the keys to represent the set. In RDF, keys are not unique, but a set can be represented by choosing a single key (a predicate) and its many values will represent the set, as a pair "key/value" is unique (aka a "predicate/object" pair). |
| <tdcolspan=3> allows concurrent edits on the value of a property that is a string. this feature is only offered by Automerge. Very useful for collaborative forms or tables/spreadsheets by example! |
| **multi-lingual strings** | ✅ 🔥 | ❌ | ❌ |
| <tdcolspan=3> Store the value of a string property in several languages / translations |
| **Counter CRDT** | ❌ | ❌ | ✅ 🔥 |
| <tdcolspan=3> Counters are a safe way to manage integers in a concurrent system. Automerge is the only one offering counters. Please note that CRDT types in general are "eventual consistent" only (BASE model). If you need stronger guarantees like the ones provided by ACID systems (specially guaranteeing the sequencing of operations, very useful for preventing double-spending) then you have to use a **Synchronous Transaction** in NextGraph. |
| <tdcolspan=3> (\*) discrete data cannot link to external documents. This is the reason why all Documents in NextGraph have a Graph part, in order to enable inter-linking of data and documents across the Giant Global Graph of Linked Data / Semantic Web |
| **Float values** | ✅ | 🟧 | ✅ |
| <tdcolspan=3> Yjs doesn't enforce strong typing on values. they can be any valid JSON (and Floats are just Numbers). |
| **Date values** | ✅ | ❌ | ✅ |
| <tdcolspan=3> JSON doesn't support JS Date datatype. So for the same reason as above, Yjs doesn't support Dates. |
| **Binary buffer values** | ✅ \* | ✅ | ✅ |
| <tdcolspan=3> (\*) as base64 or hex encoded. please note that for all purposes of storing binary data, you should use the **binary files** facility of each Document instead, which is much more efficient. |
| **boolean, integer values** | ✅ | ✅ | ✅ |
| **NULL values** | ❌ | ✅ | ✅ |
| **strongly typed decimal values** | ✅ | ❌ | ❌ |
| <tdcolspan=3> signed, unsigned, and different sizes of integers |
| **revisions, diff, revert** | 🟧 | 🟧 | 🟧 |
| <tdcolspan=3> 🔥 implemented at the NextGraph level. work in progress. You will be able to see the diffs and access all the historical data of the Document, and also revert to previous versions. |
| **compact** | ✅ | ✅ | ❓ |
| <tdcolspan=3> compacting is always available as a feature at the NextGraph level (and will compact Graph and Discrete parts alike). Yjs tends to garbage collect deleted content. not sure if automerge does it. Compact will remove all the historical data and deleted content (you won't be able to see diffs nor revert, for all the causal past happening before the compact operation. but normal CRDT behaviour can resume after) . This is a synchronous operation. |
| **snapshot** | ✅ | ✅ | ✅ |
| <tdcolspan=3> take a snapshot of the data at a given HEADs, and store it in a non-CRDT way so it can be opened quickly. Also removes all the historical and deleted data. But a snapshot cannot be used to continue collaborating on the document. See it as something similar to "export as a static file". |
| **isolated transactions** | ✅ | ✅ | ✅ |
| <tdcolspan=3> A NextGraph transaction can atomically mutate both the Graph and the Discrete data in a single isolated transaction. Can be useful to enforce consistency and keep in sync between information stored in the discrete and graph parts of the same document. but: transactions cannot span multiple documents (for that matter, see **smart contracts**). When a SPARQL Update spans across Documents, then the Transaction is split into several ones (one for each target Document) and each one is applied separately, meaning, not atomically. Also, keep in mind, as explained above in the "Counter" section, that CRDTs are eventually consistent. If you need ACID guarantees, use a synchronous transaction instead. |
| <tdcolspan=3> 🔥 this is planned. will be available shortly. the store will be **writable** and will allow a bidirectional binding of the data to some javascript reactive variables in Svelte (same could be done for React) and we are considering the use of **Valtio** for a generic reactive store, that would also work on nodeJS and Deno |
| <tdcolspan=3> (\*) support is planned at the NextGraph level, to be able to query discrete data too in SPARQL. (GraphQL support could then be added) |
| **export/import JSON** | ✅ JSON-LD | ✅ | ✅ |
| | | | |
| **Rich Text** | N/A | attributes on XMLElement | [Marks and Block Markers](https://automerge.org/docs/documents/rich_text/) and [here](https://automerge.org/docs/under-the-hood/rich_text_schema/) |
| <tdcolspan=3> Yjs integration for ProseMirror and Milkdown is quite stable. Peritext is newer and only offers ProseMirror integration. For this reason we use Yjs for Rich Text. Performance considerations should be evaluated too. |
| **Referencing rich text from outside** | N/A | ✅ 🔥 [Relative Position](https://docs.yjs.dev/api/relative-positions) | ✅ [get_cursor](https://automerge.org/automerge/automerge/trait.ReadDoc.html#tymethod.get_cursor) |
| <tdcolspan=3> useful for anchoring comments, quotes and annotations (as you shouldn't have to modify a document in order to add a comment or annotation to it). |
| **shared cursor** | N/A | 🟧 \* | 🟧 \* |
| <tdcolspan=3> (\*) available in lib but not yet integrated in NextGraph |
| <tdcolspan=3> (aka property/value, and in RDF, it is called predicate/object) this is the basic feature that the 3 models offer. You have a Document, and you can add key/value pairs to it. Also known as Map or Object. |
| <tdcolspan=3> Thanks to the Ontology/Schema mechanism of RDF (OWL), the schema information is embedded inside the data (with what we call a Predicate), thus avoiding any need for schema migration. Also, it brings full data interoperability as many common ontologies already exist and can be reused, and new ones can be published or shared |
| **nested**| ✅ blank nodes | ✅ | ✅ |
| <tdcolspan=3> key/value pairs can be nested (like in JSON) |
| **sequence**| ❌ \* | ✅ | ✅ |
| <tdcolspan=3> And like in JSON or Javascript, some keys can be Arrays (aka list), which preserve the ordering of the elements. (\*) In RDF, storing arrays is a bit more tricky. For that purpose, Collections can encode order, but are not CRDT based nor concurrency safe |
| **sets**| 🔺 multiset | ✅ | ✅ |
| <tdcolspan=3> RDF predicates (the equivalent of properties or keys in discrete documents) are not unique. They can have multiple values. That's the main difference between the discrete and graph models. We will offer the option to enforce rules on RDF data (with SHACL/SHEX) that could force unicity of keys, but that would require the use of **Synchronous Transactions**. Sets are usually represented in JS/JSON with a map, of which the values are not used (set to null or false), because keys are unique, in JSON we use the keys to represent the set. In RDF, keys are not unique, but a set can be represented by choosing a single key (a predicate) and its many values will represent the set, as a pair "key/value" is unique (aka a "predicate/object" pair). |
| <tdcolspan=3> allows concurrent edits on the value of a property that is a string. This feature is only offered by Automerge. Very useful for collaborative forms or tables/spreadsheets by example! |
| **multi-lingual strings**| ✅ 🔥 | ❌ | ❌ |
| <tdcolspan=3> Store the value of a string property in several languages / translations |
| **Counter CRDT**| ❌ | ❌ | ✅ 🔥 |
| <tdcolspan=3> Counters are a safe way to manage integers in a concurrent system. Automerge is the only one offering counters. Please note that CRDT types in general are "eventual consistent" only (BASE model). If you need stronger guarantees like the ones provided by ACID systems (specially guaranteeing the sequencing of operations, very useful for preventing double-spending) then you have to use a **Synchronous Transaction** in NextGraph. |
| <tdcolspan=3> (\*) discrete data cannot link to external documents. This is the reason why all Documents in NextGraph have a Graph part, in order to enable inter-linking of data and documents across the Giant Global Graph of Linked Data / Semantic Web |
| **Float values**| ✅ | 🟧 | ✅ |
| <tdcolspan=3> Yjs doesn't enforce strong typing on values. They can be any valid JSON (and Floats are just Numbers). |
| **Date values**| ✅ | ❌ | ✅ |
| <tdcolspan=3> JSON doesn't support JS Date datatype. So for the same reason as above, Yjs doesn't support Dates. |
| **Binary buffer values**| ✅ \* | ✅ | ✅ |
| <tdcolspan=3> (\*) as base64 or hex encoded. Please note that for all purposes of storing binary data, you should use the **binary files** facility of each Document instead, which is much more efficient. |
| **boolean, integer values**| ✅ | ✅ | ✅ |
| **NULL values**| ❌ | ✅ | ✅ |
| **strongly typed decimal values**| ✅ | ❌ | ❌ |
| <tdcolspan=3> signed, unsigned, and different sizes of integers |
| **revisions, diff, revert**| 🟧 | 🟧 | 🟧 |
| <tdcolspan=3> 🔥 implemented at the NextGraph level; work in progress. You will be able to see the diffs and access all the historical data of the Document, and also revert to previous versions. |
| **compact**| ✅ | ✅ | ❓ |
| <tdcolspan=3> compacting is always available as a feature at the NextGraph level (and will compact Graph and Discrete parts alike). Yjs tends to garbage collect deleted content; not sure if automerge does it. Compact will remove all the historical data and deleted content (you won't be able to see diffs nor revert, for all the causal past happening before the compact operation. But normal CRDT behaviour can resume after) . This is a synchronous operation. |
| **snapshot**| ✅ | ✅ | ✅ |
| <tdcolspan=3> take a snapshot of the data at a given HEADs, and store it in a non-CRDT way so it can be opened quickly. Also removes all the historical and deleted data. But a snapshot cannot be used to continue collaborating on the document. See it as something similar to "export as a static file". |
| **isolated transactions**| ✅ | ✅ | ✅ |
| <tdcolspan=3> A NextGraph transaction can atomically mutate both the Graph and the Discrete data in a single isolated transaction. Can be useful to enforce consistency and keep in sync between information stored in the discrete and graph parts of the same document. But: transactions cannot span multiple documents (for that matter, see **smart contracts**). When a SPARQL Update spans across Documents, then the Transaction is split into several ones (one for each target Document) and each one is applied separately, meaning, not atomically. Also, keep in mind, as explained above in the "Counter" section, that CRDTs are eventually consistent. If you need ACID guarantees, use a synchronous transaction instead. |
| **Svelte5 reactive Store (Runes)**| 🟧 | 🟧 | 🟧 |
| <tdcolspan=3> 🔥 this is planned and will be available shortly. The store will be **writable** and will allow a bidirectional binding of the data to some javascript reactive variables in Svelte (same could be done for React) and we are considering the use of **Valtio** for a generic reactive store, that would also work on nodeJS and Deno |
| <tdcolspan=3> (\*) support is planned at the NextGraph level, to be able to query discrete data too in SPARQL. (GraphQL support could then be added) |
| **export/import JSON**| ✅ JSON-LD | ✅ | ✅ |
| | | | |
| **Rich Text**| N/A | attributes on XMLElement | [Marks and Block Markers](https://automerge.org/docs/documents/rich_text/) and [here](https://automerge.org/docs/under-the-hood/rich_text_schema/) |
| <tdcolspan=3> Yjs integration for ProseMirror and Milkdown is quite stable. Peritext is newer and only offers ProseMirror integration. For this reason we use Yjs for Rich Text. Performance considerations should be evaluated too. |
| **Referencing rich text from outside**| N/A | ✅ 🔥 [Relative Position](https://docs.yjs.dev/api/relative-positions) | ✅ [get_cursor](https://automerge.org/automerge/automerge/trait.ReadDoc.html#tymethod.get_cursor) |
| <tdcolspan=3> useful for anchoring comments, quotes and annotations (as you shouldn't have to modify a document in order to add a comment or annotation to it). |
| **shared cursor**| N/A | 🟧 \* | 🟧 \* |
| <tdcolspan=3> (\*) available in lib but not yet integrated in NextGraph |
| **undo/redo**| N/A | 🟧 \* | 🟧 \* |
Keep on reading about how to handle the [schema](/en/framework/schema) of your data, and what the [Semantic Web](/en/framework/semantic) is all about.
@ -24,13 +24,13 @@ Those IDs and Keys will be combined in different ways detailed below, in order t
Some special format that need explanation :
- peers : a serialisation of a vector of BrokerServer that each contain the IP and port and the PeerID of the broker (and a version number). this is the “locator” part of the URI which tells which brokers should be contacted to join the overlay.
- peers : a serialisation of a vector of BrokerServer that each contain the IP and port and the PeerID of the broker (and a version number). This is the “locator” part of the URI which tells which brokers should be contacted to join the overlay.
- ReadCap : an ObjectRef (id+key) towards the current Branch definition commit
- OverlayLink: In case of an Inner Overlay, a ReadCap to the current Branch definition commit of the overlay branch of the store.
- ReadToken : is a hash of a ReadCap, it is mostly used in ExtRequests and SPARQL requests from untrusted apps or federated SPARQL requests. it gives the same level of permission than the hashed ReadCap, without revealing its content.
- ReadToken : is a hash of a ReadCap, it is mostly used in ExtRequests and SPARQL requests from untrusted apps or federated SPARQL requests. It gives the same level of permission than the hashed ReadCap, without revealing its content.
- PermaCap: a durable capability ( for read, write or query)
@ -90,11 +90,11 @@ Otherwise, without those triples, the capability is here because the repo, branc
The decomposition into several triples enables to deduplicate some information (if there are several links to the same overlay, the ng:access and ng:locator triple will be the same, so it will deduplicate) and also enables that the URI used to reference the foreign resource, will only contain the RepoId or ObjectId, which is important, specially for RepoId, as with RDF we want to establish facts between resources, that can be traversed with SPARQL queries. The unique identifier of a resource is `<did:ng:o:[repoid]>` and not the full DID capability. No matter where the resource is actually stored (the locator and overlay parts of the DID capability), which access we possess for that resource (read or write) or which revision we want to use at the moment, the unique identifier of the resource should remain the same. This is why we never use the full DID capability as URI for subject or object in RDF. Instead we use the minimal form `<did:ng:o:[repoid]>` .
We should probably mention here also that NextGraph is fully compatible with `<http(s)://…>` URIs and that they can coexist in any Document. `<ng:includes>` might work for http resources that can be dereferenced online with a GET, but we are not going to implement that right now (not a priority) and `<ng:subscribe>` will never work for those URLs, but we thought of using `<ng:refresh> "3600“ ` and `<ng:cache> “3600"` instead, which would refresh periodically the dereferenced resource in the former case, every hour, or would cache the resource after the first dereferencing for a period of 1h in the later case, but then would delete the cache to save space and would only dereference again if need be (before processing any query on the resource) and cache again for another hour. those 2 addition predicate are not planned for now as they are not a priority.
We should probably mention here also that NextGraph is fully compatible with `<http(s)://…>` URIs and that they can coexist in any Document. `<ng:includes>` might work for http resources that can be dereferenced online with a GET, but we are not going to implement that right now (not a priority) and `<ng:subscribe>` will never work for those URLs, but we thought of using `<ng:refresh> "3600“ ` and `<ng:cache> “3600"` instead, which would refresh periodically the dereferenced resource in the former case, every hour, or would cache the resource after the first dereferencing for a period of 1h in the latter case, but then would delete the cache to save space and would only dereference again if need be (before processing any query on the resource) and cache again for another hour. Those 2 additional predicate are not planned for now as they are not a priority.
### Identifiers
Here are the identifiers that can be used to refer to any resource anywhere in the graph. they are unique global identifiers.
Here are the identifiers that can be used to refer to any resource anywhere in the graph. They are unique global identifiers.
| Identifiers | that can be used in subject or object of any triple |
@ -155,7 +155,7 @@ when no default graph is included in the Query (with USING, GRAPH, FROM), the ta
| AllGroups | did:ng:g | union of all group Stores and all their documents |
| Group(String) | did:ng:g:[name] | shortname given locally to a Group (i.e. “birthday_party", “trip_to_NL") |
| | did:ng:g:g/o:[storeid] | union of all the documents of a specific group store |
| Identity(UserId) | did:ng:i:[userid] | search inside the User Storage of another Identity of the user, that is present in their wallet and has been opened. all the URI described here (except :i) can be used as suffixes behind `:i:[xxx]` |
| Identity(UserId) | did:ng:i:[userid] | search inside the User Storage of another Identity of the user, that is present in their wallet and has been opened. All the URIs described here (except :i) can be used as suffixes behind `:i:[xxx]` |
| | did:ng:i:n:[name] | same as above but with the shortname for the identity (i.e "work“ ) |
| a document | did:ng:o | |
@ -180,7 +180,7 @@ As shown above, there is a Context branch in each Document.
What is it for ?
the Context branch is an RDF mini file that contains some triples describing the JSON-LD Context. it uses the [JSON-LD Vocabulary](https://www.w3.org/ns/json-ld), and can be retrieved in its usual JSON-LD format.
the Context branch is an RDF mini file that contains some triples describing the JSON-LD Context. It uses the [JSON-LD Vocabulary](https://www.w3.org/ns/json-ld), and can be retrieved in its usual JSON-LD format.
It expresses the mapping of prefixes to the URI of the ontologies used in the document. It is common to all branches of the document. In every Header of a Document, the predicate `<ng:x>` contains as object the full link (DID cap) needed in order to open the branch and read the context. If the same context is shared across documents, then this predicate can point to a common context branch that sits outside of the document. In this case the context branch can be empty.
@ -239,5 +239,5 @@ All the files of a branch (added with the AddFile commit) will be listed by the
| ng:stores | Nuri | did:ng:o | a store stores a document<br/> (auto-generated) |
| ng:site | Nuri | did:ng:o | to link the public store from<br/>the protected store (optional) |
| ng:protected | Nuri | did:ng:o:v:r:l | link to protected store main branch,<br/> by example from a follow request |
| ng:follows | Nuri | did:ng:o | profile that is followed. needs <br/> :access, :overlay, :locator |
| ng:follows | Nuri | did:ng:o | profile that is followed. Needs <br/> :access, :overlay, :locator |
| ng:index | Nuri | did:ng:o:r | entrypoint of a service or app |
@ -14,7 +14,7 @@ Those capabilities are composed of :
- a read capability on the branch.
- If it is the root branch, then access to the whole document and all its branches will be granted.
- If it is a transactional branch (Header, Main, a block, a fork, etc...) then only this specific branch will be accessible.
- the overlay ID. see the [Network](/en/network) chapter about overlays.
- the overlay ID. See the [Network](/en/network) chapter about overlays.
- the locator which is a list of brokers that can be contacted in order to join the overlay and access the topics. Read more on topics in the [sync protocol](/en/protocol) chapter.
If the user receives write access to the repo, then the capability will contain also the ReadCap of the inner overlay.
We just finished a [roadmap](https://nextgraph.org/roadmap) of one year and half that was funded by NLnet and NGI, and that led to the publication of the first alpha release of NextGraph and all the Apps (web, linux, macOS, windows and android).
The next roadmap will be published in october 2024. We are still preparing it at the moment. It will contain several milestones related to the framework. Stay tuned!
The next roadmap will be published in early 2025. We are still preparing it at the moment. It will contain several milestones related to the framework. Stay tuned!
@ -35,7 +35,7 @@ A new ontology can be defined by creating a new Document of type Data / Ontology
There is a special prefix `ng:` for the NextGraph ontology (not to be confused with the `did:ng` method of the Nuri). This prefix is available in all RDF documents and cannot be overridden by other prefixes/context.
It has a list of predicates that help manage the Documents. It is also a way for us to offer a metadata API on each document, that can be queries with SPARQL. This API automatically generates some virtual triples about the document. let's have a look more in details about them.
It has a list of predicates that help manage the Documents. It is also a way for us to offer a metadata API on each document, that can be queries with SPARQL. This API automatically generates some virtual triples about the document. Let's have a look more in details about them.
@ -107,7 +107,7 @@ The 2 main types of queries are `SELECT` and `CONSTRUCT`. Select is similar to S
`CONSTRUCT` are a special type of queries that always return some full triples. They work the same as the SELECT WHERE, but you cannot have arbitrary variable projections. It will always return triples of the form `subject predicate object`. But you can of course tell which filters and patterns you want to follow.
Until now, we explained that each Document can hold some RDF triples. but we didn't explain how they are stored and how the SPARQL engine will be able to run queries that span all the RDF Documents that are present locally.
Until now, we explained that each Document can hold some RDF triples. But we didn't explain how they are stored and how the SPARQL engine will be able to run queries that span all the RDF Documents that are present locally.
There is in fact an option in the "SPARQL Query" tool (in the Document Menu, under "Graph" / "View as ...") that lets you query all the documents that you have present locally at once. If you do not toggle this option, you will only get results about the triples of the current Document. While with this "Query all docs" option activated, the SPARQL engine will search in all your documents, regardless if they are in the Public store, Protected store, Private store or in any Group or Dialog store.
@ -135,7 +135,7 @@ It should be noted that permissions can span a whole UserStorage, or a whole Sto
As we already mentioned shortly when we talked about blocks [here](/en/documents#blocks-and-branches), you can include some other blocks inside a document. This will have the effect of including also all the triples of such block, inside the document where the include is declared.
This is very handy when you are in Document A, and you need to access some extra data coming from another Document or Branch B, and you want to make sure that anybody who will read the current Document A, will also fetch and include the other Document or Block B automatically. If you have some logic in your document tha depends on such data, in a SPARQL query by example, this **include** mechanism will solve the headache of fetching and caching the foreign data.
This is very handy when you are in Document A, and you need to access some extra data coming from another Document or Branch B, and you want to make sure that anybody who will read the current Document A, will also fetch and include the other Document or Block B automatically. If you have some logic in your document that depends on such data, in a SPARQL query by example, this **include** mechanism will solve the headache of fetching and caching the foreign data.
The included block can be from the same document, from another document in the same store, or not, and even in someone else's document, on a remote machine. Thanks to the pub/sub and synchronization mechanism of NextGraph, this foreign block will always stay up to date as it will synchronize with the original source.
@ -143,11 +143,11 @@ The included block can be from the same document, from another document in the s
And this leads us to an explanation about what happens to named graphs in NextGraph.
Named Graphs are an RDF and SPARQL feature that lets you organize your triples into a bag. This bag contains your triples, and we call this bag a Graph. it also gets an ID in the form of a URI (URL normally. In NextGraph, a Nuri).
Named Graphs are an RDF and SPARQL feature that lets you organize your triples into a bag. This bag contains your triples, and we call this bag a Graph. It also gets an ID in the form of a URI (URL normally. In NextGraph, a Nuri).
SPARQL has options to specify which named graph(s) you want the query to relate to.
Theoretically, a triplestore can put any kind of triple in any graph. By the way, when the triplestore can understand the concept of a named graph, then we can it a quadstore, because then each triple has a 4th part telling about which graph the triple is stored in. So triples become quads (they have 4 parts instead of 3). And the triplestore becomes a quadstore.
Theoretically, a triplestore can put any kind of triple in any graph. By the way, when the triplestore can understand the concept of a named graph, then we call it a quadstore, because then each triple has a 4th part telling about which graph the triple is stored in. So triples become quads (they have 4 parts instead of 3). And the triplestore becomes a quadstore.
NextGraph is a quadstore, but there are some limitations.
@ -155,26 +155,20 @@ We do not let the user create graphs manually and arbitrarily. Instead, we assoc
Even creating a new resource or document doesn't happen freely in the quadstore. Instead, there is a special API call for creating a new document, that must be called before any triple can be inserted in this Document. This API call returns the newly generated document ID.
Then it is possible to add new triples in the this Document, and the ID of the document has to be passed to the SPARQL query, as a named graph, or as an argument in the API call itself. This way, we always know in which named graph the data should be saved or retrieved from.
Then it is possible to add new triples in the this Document, and the ID of the document has to be passed to the SPARQL query, as a named graph, or as an argument in the API call itself. This way, we always know in which named graph the data should be saved to or retrieved from.
In this Document/Named Graph, the user can add triples, and most of the time, they will add triples that have this Document ID as subject. That's what we call **authoritative triples** because we know that the subject is the same ID as the named graph, (as the document ID) and because we have signatures on every commit and also threshold signatures too, then we can prove that those triples have been authored by the users that claim to be editors of such document.
In this Document/Named Graph, the user can add triples, and most of the time, they will add triples that have this Document ID as subject, but they can also be of any other subject.
In order to facilitate adding those kind of authoritative triples with a SPARQL UPDATE, or to retrieve them with a SPARQL QUERY, the user has access to the BASE shortcut, which is `<>` in SPARQL, and that represents the current document. it will be replaced internally by the exact ID of the current document. This placeholder is handy and helps you manipulate the authoritative triples of your document. The default graph of any SPARQL Query is also the current Document, so you do not need to specify it explicitly (except when you select the "Query all docs" option, in this case, the default graph is the union graph of all the graphs).
In order to facilitate adding triples with a SPARQL UPDATE, or to retrieve them with a SPARQL QUERY, the user has access to the BASE shortcut, which is `<>` in SPARQL, and that represents the current document. It will be replaced internally by the exact ID of the current document. This placeholder is handy and helps you manipulate the triples of your document, without the new to remember the document ID. The default graph of any SPARQL Query is also the current Document, so you do not need to specify it explicitly (except when you select the "Query all docs" option, in this case, the default graph is the union graph of all the graphs).
If we had stopped here, there would be no real interest in having a named graph mechanism.
But you also are able to add triples in the Document/Named graph, that are not authoritative. Those are the triples that have as subject, some other ID than the current Document.
RDF lets anybody establish facts about any resources. If there is a foreign Document that I am using in my system, and I want to add extra information about this resource, but I don't have write permission on that foreign Document, I can add the triples in one of the Documents that I own. So it is possible to say, by example, that `London -> belongs_to -> African_continent` but of course, this is not the official point of the view of the author that manages the `London` Document. It only is "my point of view", and people who will see this triple, will also see the signature and understand that the triple is not signed by the London resource authors.
What is it useful for? RDF let's anybody establish facts about any resources. If there is a foreign Document that I am using in my system, and I want to add extra information about this resource, but I don't have write permission on that foreign Document, I can add the triples in one of the Documents that I own. External people who would see those triples that I added, would immediately understand that they are not authoritative, because they were not signed with the private key of the Document ID that they establish facts about (the subject of the triples). So it is possible to say, by example, that `London -> belongs_to -> African_continent` but of course, this is not the official point of the view of the author that manages the `London` Document. it only is "my point of view", and people who will see this triple, will also be notified that it isn't authoritative (I think they can easily understand that by themselves without the need for signatures).
Then we have other type of triples in the Document :
Then we have other use cases for extra triples in the Document :
- fragments, that are prefixed with the current document ID, and followed by a hash and a string (like in our previous example `#label`).
- fragments, that are prefixed with the authoritative ID, and followed by a hash and a string (like in our previous example `#label`).
- blank nodes that have been skolemized. They get a Nuri of the form `did:ng:o:...:u:...`. this is because blank nodes cannot exist in a local-first system, as we need to give them a unique ID. This is done with the [skolemization procedure](https://www.w3.org/TR/rdf11-concepts/#section-skolemization). For the user or programmer, skolemization is transparent. You can use blank nodes in SPARQL UPDATE and they will be automatically translated to skolems. For SPARQL QUERY anyway, blank nodes are just hidden variables, so there is no impact.
But those extra triples (fragments and skolems) are all prefixed with the authoritative ID, so they are considered authoritative too.
Note that non-authoritative triples can also have fragments and skolemized blank nodes, but their prefix will be a foreign ID, so they won't be considered authoritative neither.
- blank nodes that have been skolemized. They get a Nuri of the form `did:ng:o:...:u:...`. This is because blank nodes cannot exist in a local-first system, as we need to give them a unique ID. This is done with the [skolemization procedure](https://www.w3.org/TR/rdf11-concepts/#section-skolemization). For the user or programmer, skolemization is transparent. You can use blank nodes in SPARQL UPDATE and they will be automatically translated to skolems. For SPARQL QUERY anyway, blank nodes are just hidden variables, so there is no impact.
Now that we've explored the Semantic Web and OWL, we can dive more into [Schema definition](/en/framework/schema) in NextGraph.
@ -36,7 +36,7 @@ As a security measure, the commits and any block received on the Ext Protocol, w
This security feature is important, as external users should only be able to read what has been shared with them. If a commit has been shared, then only the transaction within the commit can be read. If a snapshot, then all the current content of the branch can be read, but it cannot be subscribed to for future updates.
If the user should be able to subscribe to the topic of the branch in order to receive updates, then a ReadCap of the branch should be share with them instead. Those branch ReadCap do not need to be signed with threshold signature because they are already signed internally with at least the first commit of the branch (the `Branch` commit) that has been signed with a synchronous signature when the branch was created. And because the user will be able to reconstruct the whole DAG up to this first commit (by doing the a sync operation) they will be able to verify the authenticity of the branch. If the authenticity of the content should also be verifiable, then an async signature must be added at the HEADs, and the user who got the branch ReadCap will get it too, and will be able to verify the authenticity of the content as well.
If the user should be able to subscribe to the topic of the branch in order to receive updates, then a ReadCap of the branch should be shared with them instead. Those branch ReadCap do not need to be signed with threshold signature because they are already signed internally with at least the first commit of the branch (the `Branch` commit) that has been signed with a synchronous signature when the branch was created. And because the user will be able to reconstruct the whole DAG up to this first commit (by doing the a sync operation) they will be able to verify the authenticity of the branch. If the authenticity of the content should also be verifiable, then an async signature must be added at the HEADs, and the user who got the branch ReadCap will get it too, and will be able to verify the authenticity of the content as well.
For now, the ext protocol and the reading of async signatures, snapshots, and commits, can be done with the CLI. Soon, APIs in the SDK will also be added.
@ -26,7 +26,7 @@ It wouldn't make sense that developers using our Framework would need to install
For this reason, we have integrated a specific feature into the Repo mechanism, that lets the developer create a Synchronous Transaction in a document (a document that should be understood as a database). Those types of transaction have higher guarantees than the Asynchronous transactions based on CRDTs, as they guarantee the **finality** of the transaction, which means that once this commit has been accepted by a certain threshold of Signers, it is guaranteed that no other concurrent transaction has happened or will ever happen that would conflict with or invalidate the invariants. Therefor the commit is final.
Basically what we do is that we temporarily prevent any fork in the DAG while the transaction is being signed (we also call that “total order”. it creates a temporary bottle neck in the DAG) and the threshold that must be above 50%, guarantees that any concurrent or future fork will be rejected. Supermajority can be used to calculate the threshold, in cases where Byzantine or faulty users have to be considered. (given N = all users, F = byzantine and/or faulty users, supermajority = (N+F)/2 +1).
Basically what we do is that we temporarily prevent any fork in the DAG while the transaction is being signed (we also call that “total order”. It creates a temporary bottle neck in the DAG) and the threshold that must be above 50%, guarantees that any concurrent or future fork will be rejected. Supermajority can be used to calculate the threshold, in cases where Byzantine or faulty users have to be considered. (given N = all users, F = byzantine and/or faulty users, supermajority = (N+F)/2 +1).
This is synchronous because it requires that the quorum of signers is online at the same time, and can agree on the transaction.
@ -42,7 +42,7 @@ You will be able to change your broker later on. Your data will move to the new
### Use the App
Once your wallet is created and you are entering the App main screen for the first time, you will have to learn how to use it. here is a brief introduction to the App.
Once your wallet is created and you are entering the App main screen for the first time, you will have to learn how to use it. Here is a brief introduction to the App.
You have created a Wallet, and inside this wallet, there is one default `Identity` that represents you. This identity is called **personal** because it represents you as an individual. You can later create more Identities if you want to separate your work life and your personal life, by example, or for other purposes of keeping your identities separated.
@ -54,7 +54,7 @@ Within each Identity, your data is organized into **3 different Stores**.
- the protected store : is a space where you can share data, documents, and media with other users, but they will need a special link and permission in order to access them and collaborate with you on those documents (or just read them). You can also create Groups here when you want to chat or collaborate with several users. This store also acts as your protected profile for social networks. Only your followers will be able to see this profile, as it is not public.
- the private store : this is a place where you put only private and personal information that only you have access to. This store will be accessible to all your devices, so it is useful in order to synchronize private data between them. You can put sensitive data too, as everything is encrypted. Nobody else will ever see this data. it is not possible to share the documents of your private store.
- the private store : this is a place where you put only private and personal information that only you have access to. This store will be accessible to all your devices, so it is useful in order to synchronize private data between them. You can put sensitive data too, as everything is encrypted. Nobody else will ever see this data. It is not possible to share the documents of your private store.
You can also later, move documents from one store to another, if you are the owner of such document.
@ -94,7 +94,7 @@ Each Document has a double nature :
- it has a _Document-like_ nature where you can store and edit some rich-text, or just some data, according to the class of the Document that you have chosen. This is so far very consistent with what we expect from a document in general. All the apps that you use with NextGraph, will store their data inside these Documents. The Document-like nature is represented with a "cloud" icon.
- A Document also has a _Graph_ nature, which is something new that NextGraph added (hence the name "NextGraph"). This "graph nature" of the document let's you link this document to other documents. It also let's you enter some specific data or meta-data about this document, that will be part of the Graph of all your documents. This graph is something important, that you are not used to. Social networks are all based on Graphs. The Web itself is a huge Graph. When you follow or get followed, when you like or comment on a Post, when you write a DM to someone, all this information is stored as a Graph, that connects different documents together. Later, you will probably need to _query_ this graph. This is done transparently when you want to see all your followers, and when you want to consult the **stream** of all the posts that they have published recently, or when you want to search for something. It is also used for recommendations. But in any case, what is important to understand is that internally, each Document can be linked to any other Document (like the classical Web that links webpages) and that the applications you will use on NextGraph, will also store more Graph information. Because NextGraph is local-first and decentralized, this **graph** information is available to you at all time. And if you want to see it, you can go to any Document and in the Document Menu, you can select "Graph" and you will see options to view and edit the Graph. This is a bit technical for now (you will see things like Turtle, SPARQL etc) but in the future, we will provide here some nice tools where you will be able to explore your own graph easily. The Graph-like nature is represented with a "sun" icon.
- A Document also has a _Graph_ nature, which is something new that NextGraph added (hence the name "NextGraph"). This "graph nature" of the document lets you link this document to other documents. It also lets you enter some specific data or meta-data about this document, that will be part of the Graph of all your documents. This graph is something important, that you are not used to. Social networks are all based on Graphs. The Web itself is a huge Graph. When you follow or get followed, when you like or comment on a Post, when you write a DM to someone, all this information is stored as a Graph, that connects different documents together. Later, you will probably need to _query_ this graph. This is done transparently when you want to see all your followers, and when you want to consult the **stream** of all the posts that they have published recently, or when you want to search for something. It is also used for recommendations. But in any case, what is important to understand is that internally, each Document can be linked to any other Document (like the classical Web that links webpages) and that the applications you will use on NextGraph, will also store more Graph information. Because NextGraph is local-first and decentralized, this **graph** information is available to you at all time. And if you want to see it, you can go to any Document and in the Document Menu, you can select "Graph" and you will see options to view and edit the Graph. This is a bit technical for now (you will see things like Turtle, SPARQL etc) but in the future, we will provide here some nice tools where you will be able to explore your own graph easily. The Graph-like nature is represented with a "sun" icon.
### Viewer and Editor
@ -125,7 +125,7 @@ From the Document Menu, you will see many other options, most of them are not im
Three menu options are implemented for now : History, "Attachments and Files" and "Tools > Signature".
The second one is, as its name indicates, about adding some images or any other attachment file to the Document. Images will be useful when you want to insert some picture in a Rich-Text document (not available yet. but you can already add pictures in the Attachments and Files).
The second one is, as its name indicates, about adding some images or any other attachment file to the Document. Images will be useful when you want to insert some picture in a Rich-Text document (not available yet but you can already add pictures in the Attachments and Files).
### History
@ -158,7 +158,7 @@ You might ask yourself what the magic carpet is.
It is not functional yet, but it will be something like a "clipboard" where you can store temporary data.
But unlike the clipboard we are used to (with CMD+C and CMD+V), the Magic Carpet can keep a list of many items, not just the latest one. This is useful when you want to move things around. Also the Magic Carpet is always reachable from anywhere in the app, and let's you drag and drop things, and see the list visually.
But unlike the clipboard we are used to (with CMD+C and CMD+V), the Magic Carpet can keep a list of many items, not just the latest one. This is useful when you want to move things around. Also the Magic Carpet is always reachable from anywhere in the app, and lets you drag and drop things, and see the list visually.
Stay tuned for more features, by checking out the [roadmap](/en/roadmap).
@ -16,7 +16,7 @@ You have heard that with NextGraph there will be no more ToS to accept. You are
#### Why is the loading so slow on the web app ?
When you use the web app, we do not store your data locally, because there is not enough space to do so in local storage (the browser limits us to 5MB). Soon we will have a new feature for the web-app, where all your data will be stored locally in the browser, thanks to indexedDB. but this is not ready yet. So for now, every time you open your wallet and start using the web app, all the document's content is fetched from the broker. This can take some time and it explains why the loading is slow. Once the documents have been retrieved, as long as you do not close the tab, we keep it locally in memory and it even works offline, in case you loose connectivity to the internet.
When you use the web app, we do not store your data locally, because there is not enough space to do so in local storage (the browser limits us to 5MB). Soon we will have a new feature for the web-app, where all your data will be stored locally in the browser, thanks to indexedDB but this is not ready yet. So for now, every time you open your wallet and start using the web app, all the document's content is fetched from the broker. This can take some time and it explains why the loading is slow. Once the documents have been retrieved, as long as you do not close the tab, we keep it locally in memory and it even works offline, in case you loose connectivity to the internet.
#### Where is my data stored when I install the native App ?
@ -25,7 +25,7 @@ When you use our native app, all the content of the documents is stored locally
- on Linux: `~/.local/share/org.nextgraph.app`
- on macOS: `/Users/[username]/Library/Application Support/org.nextgraph.app`
- on Windows: `C:\Users\[username]\AppData\Roaming\org.nextgraph.app`
- on Android and iOS: the folder is not accessible to you. it is protected by the OS.
- on Android and iOS: the folder is not accessible to you. It is protected by the OS.
You will soon be able to use the CLI in order to read your local data (on Desktop and Laptop).
@ -33,11 +33,11 @@ We recommend you to **not configure** your mobile NextGraph app for Cloud backup
#### I cannot share my documents with other users
This is a temporary limitation that will be lifted in several weeks from now (in the course of Q3 2024).
This is a temporary limitation that will be lifted in several weeks from now (in the course of Q1 2025).
#### Can I create several Wallet for myself ?
We do not recommend you to create more than one Wallet. The app is not designed to handle such case. Instead, the app has been designed so that you can create more separate identities from within the same wallet. identities that are stored in the same wallet are still independent one from another and nobody will be able to tell that you own both identities. This feature is not available yet but will come soon. There is really no need to create several wallets. Specially because you will have difficulties to remember several pazzles.
We do not recommend you to create more than one Wallet. The app is not designed to handle such case. Instead, the app has been designed so that you can create more separate identities from within the same wallet. Identities that are stored in the same wallet are still independent one from another and nobody will be able to tell that you own both identities. This feature is not available yet but will come soon. There is really no need to create several wallets. Specially because you will have difficulties to remember several pazzles.
@ -16,7 +16,7 @@ In order to work properly, a Local First app needs to use CRDTs (Conflict-free R
Some rules are agreed upon in advance on how to deal with the conflict, based on the metadata. And those rules lead to a deterministic and consistent conflict resolution mechanism across replicas, regardless of the order in which the updates are applied.
The best CRDTs out there are based on a DAG of dependencies, that encodes the causal past of an update/operation/commit (all synonyms).
The best CRDTs out there are based on a DAG of dependencies (Directed Acyclic Graph), that encodes the causal past of an update/operation/commit (all synonyms).
Each operation indicates which previous operations it “sees” in its “causal past” at the moment when the operation is committed.
@ -30,7 +30,7 @@ To come back to the higher level overview, local-first apps deal well with offli
Be it a document that is shared between the different devices of a single user, or a document that is shared among several users (and their respective many devices), the user will always be able to view and/or edit the document while being offline, and then sync with the other replicas after regaining connectivity.
This by itself, is a big paradigm shift for the developers, who is more used to calling some remote APIs for accessing the data.
This by itself, is a big paradigm shift for the developers, who are more used to calling remote APIs to access data.
Here the data is always accessed locally (in this sense, the backend sits in the “client” or in the “front-end" if you prefer).
@ -90,7 +90,7 @@ We also have a pair of overlay for each Group, and only one Inner overlay for ea
The Outer overlay ID is just the Blake3 hash of the Store ID.
While the Inner overlay ID is a Blake3 keyed hash of the Store ID. the key used for the hash is derived from the ReadCapSecret of the Overlay branch of the Store.
While the Inner overlay ID is a Blake3 keyed hash of the Store ID. The key used for the hash is derived from the ReadCapSecret of the Overlay branch of the Store.
We will now dive more into details about repositories, branches, blocks and commits, that compose the [NextGraph Protocol](/en/protocol)
@ -32,7 +32,7 @@ This new commit is now the current head at the local replica, and will be sent t
It can happen that other editors make concurrent modifications. In this case, they will also publish a commit with a causal past (ACKS) that is similar or identical to the new commit we just published.
This will lead to a temporary “fork” in the DAG, and after the replicas have finished their syncing, they will all have 2 current heads. one for each of the concurrent commits.
This will lead to a temporary “fork” in the DAG, and after the replicas have finished their syncing, they will all have 2 current heads, one for each of the concurrent commits.
The next commit (whoever will make more modification in the document), will “merge” the fork when it will publish a new commit that references the 2 heads as ACKS (direct causal past).
@ -56,7 +56,7 @@ A Repo (repository) is a unit that regroups one or several branches of content,
When a repo is created, it comes with 2 branches by default :
- the root branch, which is used to store all the members information, their permissions, the list of branches, and controls the epochs. it does not hold content. Its branchID cannot change because it is in fact, the same ID as the RepoID itself.
- the root branch, which is used to store all the members information, their permissions, the list of branches, and controls the epochs. It does not hold content. Its branchID cannot change because it is in fact, the same ID as the RepoID itself.
- the main branch, which is a transactional branch (transactional=that holds content) and that will be the default branch if no other branch is specified when a request is made to access the content. It is possible to change the main branch and point it to another branch in the repo.
@ -70,7 +70,7 @@ It is possible to renew the topicID during the lifetime of a branch, even severa
This renew mechanism is used when the capabilities of the branch needs to be refreshed (for read access, when we want to remove read access from some user).
The write access is not controlled by branch, but is controlled more generally at the repo level. it is not possible to give write permission only to one specific branch. When a member is given write permission, it applies to all the branches of the repo at once. The same when write permission is revoked. It is revoked for all the branches of the repo at once.
The write access is not controlled by branch, but is controlled more generally at the repo level. It is not possible to give write permission only to one specific branch. When a member is given write permission, it applies to all the branches of the repo at once. The same when write permission is revoked. It is revoked for all the branches of the repo at once.
It is indeed important that permissions are common to all branches, because we will now see that branches can be merged one into another. And when the merge happens, we consider that all the commits of the branches are valid and have been verified already back then, at the moment of every commit addition. We do not want to have to re-verify a whole branch before it is merged. What was already verified and accepted, is immutably part of the repo. If we had a permission system with different permissions for each branch, then there would be cases when some commits in one branch, cannot be merged into another branch because the permissions are incompatible. In order to prevent this, and also to simplify an already very complex design, we restricted the permission management to be only at the repo level, unlike the previous design of LoFi.Re.
@ -92,11 +92,11 @@ This is due to the fact that a read permission is in fact a cryptographic capabi
That's very handy, if we want to separate a Document into several parts that will have different read access.
Let's say I have a Document that is my personal profile description. it contains my pseudonym, full name, date of birth, postal address, email address, phone number, short biography, profile picture, etc…
Let's say I have a Document that is my personal profile description. It contains my pseudonym, full name, date of birth, postal address, email address, phone number, short biography, profile picture, etc…
Now let's imagine that for some reasons related to my privacy, I do not always want to share my postal address and phone number with everyone, but instead I want to opt-out sometimes and share the rest, but not the postal address and phone number.
I could create two different documents. one with all the info, and one with the reduced profile.
I could create two different documents. One with all the info, and one with the reduced profile.
But that would be cumbersome, as every time I need to update my bio, by example, i would have to copy paste it in both Documents.
@ -106,11 +106,11 @@ Both branches are updatable. If I modify my bio, all the users who subscribed to
And I can even include a link to the main branch, from within the privateProfile branch, so that those trusted people also have access to the main branch, without need for me to share both branches ReadCaps with them.
If at some point in the future, I want to merge those two branches into one, well.. that, I won't be able to do it, because in order to merge two branches, they need to share a common ancestor (one branch has to be a fork from the other).
If at some point in the future, I want to merge those two branches into one, well.. That, I won't be able to do it, because in order to merge two branches, they need to share a common ancestor (one branch has to be a fork from the other).
But here, those 2 branches are completely separated one from another. The only thing they share is that they belong to the same Repo, but they both have zero ancestors in their root DAG commit. those 2 DAGs are unrelated one to another. So we cannot merge them.
But here, those 2 branches are completely separated one from another. The only thing they share is that they belong to the same Repo, but they both have zero ancestors in their root DAG commit. Those 2 DAGs are unrelated one to another. So we cannot merge them.
Another example about how we can use branches to do cool stuff, is for commenting/annotating on someone else's content. Commenting is a kind of editing, as it adds content. But we don't want to have to invite those commentators as editors of the document they want to comment on. Instead the commentator will create a standalone branch somewhere on their own protected store (they are free to proceed as they want on that. They can create a special document on their side, that will have the sole purpose of holding all the branches used for each comment on a specific target Document. or they can even use less Documents, and have one general purpose Document in their protected store that is always used to create branches for commenting, regardless of the target document that is commented upon.) What matters is that they are the only editor on that Document, and they will write one comment by branch. The branch subscription mechanism will let them update/fix typos on that specific comment later on. They can also delete the branch at any time, in order to delete their own comment. Once they have created that branch and inserted some content in it (the comment itself), they will send a link (a DID cap) to the original Document they want to comment upon. (each document has an inbox, which is used in this case to drop the link). A comment can reference previous comment, or quote some part of the document (annotation), thanks to RDF, this is easy to do. The owner of the Document that receives this link that contains a comment, can moderate it, accept, reject, or remove it after accepting it. If accepted, the link (DID cap) is added a the special branch for comments, that every document has by default (more on that below). Any reader of the document that subscribed to this branch, will see the new comment.
Another example about how we can use branches to do cool stuff, is for commenting/annotating on someone else's content. Commenting is a kind of editing, as it adds content. But we don't want to have to invite those commentators as editors of the document they want to comment on. Instead the commentator will create a standalone branch somewhere on their own protected store (they are free to proceed as they want on that. They can create a special document on their side, that will have the sole purpose of holding all the branches used for each comment on a specific target Document. Or they can even use less Documents, and have one general purpose Document in their protected store that is always used to create branches for commenting, regardless of the target document that is commented upon.) What matters is that they are the only editor on that Document, and they will write one comment by branch. The branch subscription mechanism will let them update/fix typos on that specific comment later on. They can also delete the branch at any time, in order to delete their own comment. Once they have created that branch and inserted some content in it (the comment itself), they will send a link (a DID cap) to the original Document they want to comment upon. (each document has an inbox, which is used in this case to drop the link). A comment can reference previous comment, or quote some part of the document (annotation), thanks to RDF, this is easy to do. The owner of the Document that receives this link that contains a comment, can moderate it, accept, reject, or remove it after accepting it. If accepted, the link (DID cap) is added a the special branch for comments, that every document has by default (more on that below). Any reader of the document that subscribed to this branch, will see the new comment.
So, to recap.
@ -122,28 +122,28 @@ So, to recap.
- i can also fork a branch into another branch, and then merge that fork back into the original branch (or into any other branch that shares a common ancestor)
- those forks can be used to store some specific revisions of the document. and then, by using the branchId, it is possible to refer to that specific revision.
- those forks can be used to store some specific revisions of the document. And then, by using the branchId, it is possible to refer to that specific revision.
- a branch can also be given a name, like "rewriting_paragraph_B“.
- any given commit has an ID, and that commit can also be used to refer to a specific revision, which in this case, is just the state of the document at that very specific commit. commits can also be given names like v0_1_0 (equivalent to the tags in GIT), and those names are pointers that can be updated. so one can share the name, and update the pointer later on.
- any given commit has an ID, and that commit can also be used to refer to a specific revision, which in this case, is just the state of the document at that very specific commit. Commits can also be given names like v0_1_0 (equivalent to the tags in GIT), and those names are pointers that can be updated. So one can share the name, and update the pointer later on.
- standalone branches can be used to separate different segments of data that need different read permissions.
- ReadCaps can be refreshed in order to remove read access to some branch (but the historical data they used to have access to, will always remain visible to them, specially because everything is local-first, so they surely have a local copy of that historical data. what they won't see are the new updates).
- ReadCaps can be refreshed in order to remove read access to some branch (but the historical data they used to have access to, will always remain visible to them, specially because everything is local-first, so they surely have a local copy of that historical data. What they won't see are the new updates).
- we use the terms “DID cap”, "ReadCap", “URI", “link” or “[Nuri](/en/framework/nuri)” interchangeably in this document. They all mean the same.
it is also possible to fork a whole repo, if ownership and permissions need to be changed (similar to the “fork me on github” feature) and then there is a mechanism for “pull requests” in order to merge back that forked repo into the original repo. But it doesn't work like merging of branches, as each commit has to be checked again separately and added to the DAG again, using the identity of a user that has write permission in the target repo. Let's leave that for now, as it is not coded yet, and not urgent.
The root branch is a bit complex and has all kind of system commits to handle the internals of permissions etc. We will not dive into that right now. There are also some other hidden system branches (called Store, User, Overlay, Chat, etc..) that contain some internal data used by the system, and that you can imagine a bit what it does, given the reserved names they have. but again, let's keep that for later.
The root branch is a bit complex and has all kind of system commits to handle the internals of permissions etc. We will not dive into that right now. There are also some other hidden system branches (called Store, User, Overlay, Chat, etc..) that contain some internal data used by the system, and that you can imagine a bit what it does, given the reserved names they have. But again, let's keep that for later.
What matters for now is that any transactional branch contains commits that modify the content of the branch, which is a revision of the document.
Those commits are encrypted and sent as events in the pub/sub.
When a commit arrives on a replica, the Verifier is in charge of verifying the integrity of the commit and the branches and repo in general, and this Verifier will need to read the ACLs. it will also verify some signatures and do some checks on the DAG.
When a commit arrives on a replica, the Verifier is in charge of verifying the integrity of the commit and the branches and repo in general, and this Verifier will need to read the ACLs. It will also verify some signatures and do some checks on the DAG.
If something goes wrong, the commit is rejected and discarded. its content is not passed to the application level.
If something goes wrong, the commit is rejected and discarded. Its content is not passed to the application level.
Eventually, all the replicas have a local set of commits for a branch, and they need to read them and process them once, in order to build the materialized state of the doc. That's the job of the [verifier](/en/verifier).
@ -4,7 +4,7 @@ description: Self host NextGraph broker on your own server or locally
layout: ../../layouts/MainLayout.astro
---
Self-hosting of the broker will come at the end of 2024. Stay tuned for more features and added freedom!
Self-hosting of the broker will come in 2025. Stay tuned for more features and added freedom!
When you self-host a broker, you enjoy the exact same features than with our public Broker Service Providers, but in addition, you don't have to abide by any Terms of Services, and your data is free from any limitations.
@ -4,7 +4,7 @@ description: NextGraph aims to offer a robust, private, secure and easy to use S
layout: ../../layouts/MainLayout.astro
---
We have seen that NextGraph is [organized around documents](/en/documents), and that we can share and collaborate on the those documents with other users.
We have seen that NextGraph is [organized around documents](/en/documents), and that we can share and collaborate on those documents with other users.
The internal mechanism of synchronization of the documents is based on a pub/sub, and this enables us to offer all the range of features and services that a Social Network offers.
@ -22,13 +22,13 @@ The only part that we had to design and implement ourselves, is the [network pro
As we also know that some existing protocols are widely used, we will strive to be compatible with them.
ActivityPub is the most famous OSS protocol for social network, and we have started a collaboration with the [ActivityPods project](https://activitypods.org) in order to bring full compatibility of NextGraph with ActivityPub protocol, and also with the Solid standard! We also aim at harmonizing our frameworks so that apps developed for ActivityPods will also work on NextGraph framework and thus benefit from local-first and end-to-end encryption, and the reverse will also be true. Apps developed on NextGraph framework will also work on ActivityPods, and therefor be compatible with ActivityPub and Solid.
ActivityPub is the most famous OSS protocol for social network, and we have started a collaboration with the [ActivityPods project](https://activitypods.org) in order to bring full compatibility of NextGraph with ActivityPub protocol, and also with the Solid standard! We also aim at harmonizing our frameworks so that apps developed for ActivityPods will also work on NextGraph framework and thus benefit from local-first and end-to-end encryption, and the reverse will also be true. Apps developed on NextGraph framework will also work on ActivityPods, and therefore be compatible with ActivityPub and Solid.
ActivityPods will act as a gateway between the world of HTTP and NextGraph, that has severed ties with HTTP for reasons explained in the [Encryption chapter](/en/encryption).
Later on, we also aim at being compatible with Nostr, BlueSky/ATproto and maybe more protocols too.
We believe that offering Social Network features in the heart of our framework is essential. here is a list of the features that will soon be part of our framework and platform.
We believe that offering Social Network features in the heart of our framework is essential. Here is a list of the features that will soon be part of our framework and platform.
### Stream
@ -70,7 +70,7 @@ But this has a negative side-effect. The global graph that Meta by example, main
We will overcome this caveat of not having access to the global graph, by enabling peer-to-peer traversal of the graph.
This will be possible very soon in NextGraph, thanks to the use of the Semantic Web and the features of "federated queries" that let's you query the data that is not on your machine, but instead, sits in the machine of your friends and contacts. If those other users give you the permission to do so, you will be able to search in their own graph too.
This will be possible very soon in NextGraph, thanks to the use of the Semantic Web and the features of "federated queries" that lets you query the data that is not on your machine, but instead, sits in the machine of your friends and contacts. If those other users give you the permission to do so, you will be able to search in their own graph too.
This is why NextGraph makes a strong separation between the data that is private and should always stay private, and the data that is public or protected. Public data can be queried by anyone, while protected data needs your permission.
@ -98,8 +98,8 @@ Maybe one day the European Union will finally force them to open their APIs and
In the meanwhile, we are currently exploring ways to implement this already.
Another important tool would be the option to import all the content and contacts that someone has gathered through the years on a platform of big-tech, so it can be used in NextGraph, when that person decides to switch to our platform. Some big-tech platforms already offer the export option. and we would have to implement the import part.
Another important tool would be the option to import all the content and contacts that someone has gathered through the years on a platform of big-tech, so it can be used in NextGraph, when that person decides to switch to our platform. Some big-tech platforms already offer the export option, and we would have to implement the import part.
It is important to understand that NextGraph does not need to implement an export feature, because with NextGraph, all your data is totally under your control already. it is local and you have it directly on your device, it is saved in some open standards like RDF, JSON or Markdown, and you can do whatever you want with it. Furthermore, if you decide at some point to use a different broker, or to self-host your own broker, all your data at NextGraph is absolutely **portable** and you can move your data to another broker without any bad consequence. Your documents and links have unique IDs that are not related to the broker where you store the data. That's another advantage that no other platform or framework can offer, not even ActivityPub or Solid, which are all based on http and domain names, and that will break as soon as you change hosting provider. (I have experienced this myself. In Mastodon, the list of followers can easily be transferred, but you loose all your posts and DMs).
It is important to understand that NextGraph does not need to implement an export feature, because with NextGraph, all your data is totally under your control already. It is local and you have it directly on your device, it is saved in some open standards like RDF, JSON or Markdown, and you can do whatever you want with it. Furthermore, if you decide at some point to use a different broker, or to self-host your own broker, all your data at NextGraph is absolutely **portable** and you can move your data to another broker without any bad consequence. Your documents and links have unique IDs that are not related to the broker where you store the data. That's another advantage that no other platform or framework can offer, not even ActivityPub or Solid, which are all based on http and domain names, and that will break as soon as you change hosting provider. (I have experienced this myself. In Mastodon, the list of followers can easily be transferred, but you loose all your posts and DMs).
Welcome to NextGraph, that will soon offer all those Social Network features, and that already offers the best [collaborative tools](/en/collaboration) with offline-first and end-to-end encryption.
@ -4,6 +4,6 @@ description: NextGraph is compatible with the Solid Platform and Protocols, than
layout: ../../layouts/MainLayout.astro
---
Thanks to our collaboration with the [ActivityPods project](https://activitypods.org) we will soon offer full compatibility with the [Solid standard](https://solidproject.org/). At the same time, all apps developed on ActivityPods framework will be compatible with NextGraph.
Thanks to our collaboration with the [ActivityPods project](https://activitypods.org/activitypods-and-nextgraph-are-joining-forces) we will soon offer full compatibility with the [Solid standard](https://solidproject.org/). At the same time, all apps developed on ActivityPods framework will be compatible with NextGraph.
/// BEC periodic reconciliation interval. zero deactivates it
/// BEC periodic reconciliation interval, zero deactivates it
reconciliation_interval: RelTime,
/// list of owners
@ -108,7 +108,7 @@ struct RootBranchV0 {
- owners : all of them are required to sign any RootBranch that modifies the list of owners or the inherit_perms_users_and_quorum_from_store field.
- owners_write_cap : when the list of owners is changed, a crypto_box containing the RepoWriteCapSecret should be included here for each owner. This should also be done at creation time, with the UserId of the first owner, except for individual private store repo, because it doesn't have a RepoWriteCapSecret. The vector has the same order and size as the owners one. each owner finds their write_cap here.
- owners_write_cap : when the list of owners is changed, a crypto_box containing the RepoWriteCapSecret should be included here for each owner. This should also be done at creation time, with the UserId of the first owner, except for individual private store repo, because it doesn't have a RepoWriteCapSecret. The vector has the same order and size as the owners one. Each owner finds their write_cap here.
### Branch
@ -152,7 +152,7 @@ struct BranchV0 {
- topic_privkey : a BranchWriteCapSecret, encrypted with a nonce = 0 and a key derived as follow
BLAKE3 derive_key ("NextGraph Branch WriteCap Secret BLAKE3 key", RepoWriteCapSecret, TopicId, BranchId ) so that only editors of the repo can decrypt the privkey. For individual store repo, the RepoWriteCapSecret is zero
- pulled_from : optional: this branch is the result of a pull request coming from another repo. contains a serialization of a ReadBranchLink of a transactional branch from another repo
- pulled_from : optional: this branch is the result of a pull request coming from another repo. Contains a serialization of a ReadBranchLink of a transactional branch from another repo
### AddBranch
@ -183,7 +183,7 @@ struct AddBranchV0 {
}
```
- topic_id : the new topic_id. Will be needed immediately by future readers in order to subscribe to the pub/sub). should be identical to the one in the Branch definition. None if merged_in.
- topic_id : the new topic_id. Will be needed immediately by future readers in order to subscribe to the pub/sub). Should be identical to the one in the Branch definition. None if merged_in.
- branch_read_cap : the new branch definition commit (we need the ObjectKey in order to open the pub/sub Event). None if merged_in
@ -218,7 +218,7 @@ Points to the new Signature Object. Based on the total order quorum (or owners q
**DEPS**: the last signed commit in chain
**ACKS**: previous head before the chain of signed commit(s). should be identical to the HEADS (marked as DEPS) of first commit in chain
**ACKS**: previous head before the chain of signed commit(s). Should be identical to the HEADS (marked as DEPS) of first commit in chain
```rust
enum SyncSignature {
@ -254,7 +254,7 @@ struct SignatureV0 {
/// the content that is signed
content: SignatureContent,
/// The threshold signature itself. can come from 3 different sets
/// The threshold signature itself. Can come from 3 different sets
threshold_sig: ThresholdSignatureV0,
/// Reference to the Certificate that must be used to verify this signature.
@ -272,7 +272,7 @@ struct SignatureContentV0 {
commits: Vec<ObjectId>,
}
// the threshold signature itself. with indication which set was used
// the threshold signature itself, with indication which set was used
enum ThresholdSignatureV0 {
PartialOrder(threshold_crypto::Signature),
TotalOrder(threshold_crypto::Signature),
@ -322,7 +322,7 @@ struct CertificateContentV0 {
readcap_id: ObjectId,
/// PublicKey used by the Owners. verifier uses this PK
/// PublicKey used by the Owners. Verifier uses this PK
// if the signature was issued by the Owners.
owners_pk_set: threshold_crypto::PublicKey,
@ -348,7 +348,7 @@ enum OrdersPublicKeySetsV0 {
- CertificateSignatureV0.Owners : indicates that the owners set signed the certificate. If the previous cert's total order PKset has a threshold value of 0 or 1 (1 or 2 signers in the quorum),then it is allowed that the next certificate (this one) will be signed by the owners PKset instead. This is for a simple reason: if a user is removed from the list of signers in the total_order quorum,then in those 2 cases, the excluded signer will probably not cooperate to their exclusion, and will not sign the new certificate. To avoid deadlocks, we allow the owners to step in and sign the new cert instead. The Owners are also used when there is no quorum/signer defined (OrdersPublicKeySetsV0::None).
- CertificateSignatureV0.Store : in case the new certificate being signed is an update on the store certificate (OrdersPublicKeySetsV0::Store(ObjectRef) has changed from previous cert) then the signature is in that new store certificate, and not here. nothing else should have changed in the CertificateContent, and the validity of the new store cert has to be checked.
- CertificateSignatureV0.Store : in case the new certificate being signed is an update on the store certificate (OrdersPublicKeySetsV0::Store(ObjectRef) has changed from previous cert) then the signature is in that new store certificate, and not here. Nothing else should have changed in the CertificateContent, and the validity of the new store cert has to be checked.
- CertificateContentV0.previous : the previous certificate in the chain of trust. Can be another Certificate or the Repository commit's body when we are at the root of the chain of trust.
@ -359,7 +359,7 @@ enum OrdersPublicKeySetsV0 {
- OrdersPublicKeySetsV0::Repo.0 one for the total_order (first one).
- OrdersPublicKeySetsV0::Repo.1 the other for the partial_order (second one.is optional, as some repos are forcefully totally ordered and do not have this set).
- OrdersPublicKeySetsV0::None : the total_order quorum is not defined (yet, or anymore). there are no signers for the total_order, neither for the partial_order. The owners replace them.
- OrdersPublicKeySetsV0::None : the total_order quorum is not defined (yet, or anymore). There are no signers for the total_order, neither for the partial_order. The owners replace them.
### StoreUpdate
@ -389,7 +389,7 @@ Adds a repo into the store branch.
The repo's `store` field should match the destination store
**DEPS**: to the previous AddRepo commit(s) if it is an update. in this case, repo_id of the referenced rootbranch definition(s) should match
**DEPS**: to the previous AddRepo commit(s) if it is an update. In this case, repo_id of the referenced rootbranch definition(s) should match
```rust
struct AddRepoV0 {
@ -408,7 +408,7 @@ So that a user can share with all its device a new signing capability that was j
The cap's `epoch` field should be dereferenced and the user must be part of the quorum/owners.
**DEPS**: to the previous AddSignerCap commit(s) if it is an update. in this case, repo_ids have to match,
**DEPS**: to the previous AddSignerCap commit(s) if it is an update. In this case, repo_ids have to match,
and the referenced rootbranch definition(s) should have compatible causal past (the newer AddSignerCap must have a newer epoch compared to the one of the replaced cap )
```rust
@ -485,7 +485,7 @@ Add a new binary file in a branch
```rust
struct AddFileV0 {
/// an optional name. does not conflict
/// an optional name, does not conflict
/// (not unique across the branch nor repo)
name: Option<String>,
@ -533,7 +533,7 @@ struct CommitV0 {
/// Commit content
content: CommitContent,
/// Signature over the content by the author. an editor (UserId)
/// Signature over the content by the author; an editor (UserId)
sig: Sig,
}
@ -588,7 +588,7 @@ struct CommitHeaderKeysV0 {
CommitContentV0:
- author : Commit author, a BLAKE3 keyed hash of UserId. key is a BLAKE3 derive_key ("NextGraph UserId Hash Overlay Id for Commit BLAKE3 key", overlayId). Hash will be different than for ForwardedPeerAdvertV0 so that core brokers dealing with public sites wont be able to correlate commits and editing peers (via common author's hash).O nly the brokers of the authors that pin a repo for Outer Overlay exposure, will be able to correlate. It also is a different hash than the OuterOverlayId, which is good to prevent correlation when the RepoId is used as author (for Repository, RootBranch and Branch commits)
- author : Commit author, a BLAKE3 keyed hash of UserId. Key is a BLAKE3 derive_key ("NextGraph UserId Hash Overlay Id for Commit BLAKE3 key", overlayId). Hash will be different than for ForwardedPeerAdvertV0 so that core brokers dealing with public sites wont be able to correlate commits and editing peers (via common author's hash).O nly the brokers of the authors that pin a repo for Outer Overlay exposure, will be able to correlate. It also is a different hash than the OuterOverlayId, which is good to prevent correlation when the RepoId is used as author (for Repository, RootBranch and Branch commits)
- branch : BranchId the commit belongs to (not a ref, as readers do not need to access the branch definition)
@ -627,7 +627,7 @@ struct CommitHeaderV0 {
/// Other objects this commit strongly depends on
deps: Vec<ObjectId>,
/// dependency that is removed after this commit. used for reverts
/// dependency that is removed after this commit; used for reverts
ndeps: Vec<ObjectId>,
compact: bool,
@ -644,7 +644,7 @@ struct CommitHeaderV0 {
}
```
- compact : tells brokers that this is a hard snapshot and that all the ACKs and full causal past should be treated as ndeps (their body removed). brokers will only perform the deletion of bodies after this commit has been ACKed by at least one subsequent commit. but if the next commit is a nack, the deletion is aborted.
- compact : tells brokers that this is a hard snapshot and that all the ACKs and full causal past should be treated as ndeps (their body removed). Brokers will only perform the deletion of bodies after this commit has been ACKed by at least one subsequent commit. But if the next commit is a nack, the deletion is aborted.
### RandomAccessFile
@ -749,20 +749,20 @@ type InternalNode = Vec<BlockKey>
```
- BlockV0.commit_header_key : optional Key needed to open the CommitHeader. can be omitted if the Commit is shared without its causal past, or if the block is not a root block of commit, or that commit is a root commit (first in branch)
- BlockV0.commit_header_key : optional Key needed to open the CommitHeader. Can be omitted if the Commit is shared without its causal past, or if the block is not a root block of commit, or that commit is a root commit (first in branch)
- BlockContentV0.commit_header : Reference (actually, only its ID or an embedded block if the size is small enough)
to a CommitHeader of the root Block of a commit that contains references to other objects (e.g. Commit deps & acks).
Only set if the block is a commit (and it is the root block of the Object).
It is an easy way to know if the Block is a commit (but be careful because some root commits can be without a header).
- BlockContentV0.children : Block IDs for child nodes in the Merkle tree. It is empty if ObjectContent fits in one block or this block is a leaf. in both cases, encrypted_content is then not empty
- BlockContentV0.children : Block IDs for child nodes in the Merkle tree. It is empty if ObjectContent fits in one block or this block is a leaf. In both cases, encrypted_content is then not empty
- BlockContentV0.encrypted_content : contains an encrypted ChunkContentV0, encrypted using convergent encryption with ChaCha20: nonce = 0 and key = BLAKE3 keyed hash (convergence_key, plaintext of ChunkContentV0), with convergence_key = BLAKE3 derive_key ("NextGraph Data BLAKE3 key", StoreRepo + store's repo ReadCapSecret ) which is basically similar to the InnerOverlayId but not hashed, so that brokers cannot do "confirmation of a file" attacks.
## Event
An event is a commit with some additional meta-data needed for sending it into the pub/dub topic.
An event is a commit with some additional meta-data needed for sending it into the pub/sub topic.
It can also contain additional blocks that are sent together with the commit (additional_blocks).
**All our protocols and formats use the binary codec called [BARE](https://baremessages.org/)**.
The App Protocol let's the Application talk with the Verifier.
The App Protocol lets the Application talk with the Verifier.
This protocol exchanges content that isn't encrypted.
@ -194,7 +194,7 @@ enum BrokerServerTypeV0 {
- NuriV0.identity : None for personal identity
- NuriV0.entire_store : If it is a store, will include all the docs belonging to the store. not used otherwise
- NuriV0.entire_store : If it is a store, will include all the docs belonging to the store. Not used otherwise
- NuriV0.objects : used only for FileGet.
@ -469,7 +469,7 @@ struct DocAddFile {
- DocAddFile.object : must be the reference you obtained from the last call to `RandomAccessFilePutChunk`.
- DocAddFile.filename : an optional filename. usually it is the original filename on the filesystem when selecting the binary file to upload.
- DocAddFile.filename : an optional filename. Usually it is the original filename on the filesystem when selecting the binary file to upload.
#### final Response
@ -733,7 +733,7 @@ enum DocQuery {
}
```
- AppRequestV0.nuri.target : represents the default graph. can be NuriV0::UserSite or NuriV0::None and in those cases, the **union graph** of all the graphs is used as default graph.
- AppRequestV0.nuri.target : represents the default graph. Can be NuriV0::UserSite or NuriV0::None and in those cases, the **union graph** of all the graphs is used as default graph.
- DocQuery::V0.base : an optional base to resolve all your relative URIs of resources in the SPARQL Query.
@ -743,7 +743,7 @@ enum DocQuery {
Depending on the type of query, the response differs.
- for SELECT queries: an `AppResponseV0::QueryResult(buffer)` where buffer is a `Vec<u8>` containing a UTF-8 serialization of a JSON string representing a JSON Sparql Query Result. see [SPARQL Query Results JSON Format](https://www.w3.org/TR/sparql11-results-json/).
- for SELECT queries: an `AppResponseV0::QueryResult(buffer)` where buffer is a `Vec<u8>` containing a UTF-8 serialization of a JSON string representing a JSON Sparql Query Result. See [SPARQL Query Results JSON Format](https://www.w3.org/TR/sparql11-results-json/).
- for CONSTRUCT an `AppResponseV0::Graph(buffer)` where buffer is a `Vec<u8>` containing a BARE serialization of a `Vec<Triple>`.
@ -787,7 +787,7 @@ AppRequestV0 {
nuri: NuriV0 {
target: NuriTargetV0::Repo(repo_id),
overlay: Some(overlay_id),
branch: //not implemented. defaults to main branch
branch: //not implemented. Defaults to main branch
... // all the rest empty
},
payload: None
@ -820,7 +820,7 @@ struct CommitInfo {
- AppHistory.history : in order, with the newest commits first. The first part of the tuple (ObjectId) is the CommitID.
- AppHistory.swimlane_state : can be discarded. it is only used by our GUI representation of the history in the Apps. Same with the x and y values in CommitInfo.
- AppHistory.swimlane_state : can be discarded. It is only used by our GUI representation of the history in the Apps. Same with the x and y values in CommitInfo.
### SignatureStatus
@ -836,7 +836,7 @@ AppRequestV0 {
nuri: NuriV0 {
target: NuriTargetV0::Repo(repo_id),
overlay: Some(overlay_id),
branch: //not implemented. defaults to main branch
branch: //not implemented. Defaults to main branch
... // all the rest empty
},
payload: None
@ -850,7 +850,7 @@ The response is a `AppResponseV0::SignatureStatus(Vec<(String, Option<String>, b
Which is a list of commits at the HEAD. Each commit is represented by a tuple that contains :
- the commit ID printed as a string (44 characters)
- an optional string that is present only if the commit is a signature. in this case, the string represents a list of the signed commits `c:[commit_id]:k:[commit_key]` joined with `:` (at least one commit is present), followed by partial Nuri for the signature object `:s:[signature_object_id]:k:[signature_object_key]`.
- an optional string that is present only if the commit is a signature. In this case, the string represents a list of the signed commits `c:[commit_id]:k:[commit_key]` joined with `:` (at least one commit is present), followed by partial Nuri for the signature object `:s:[signature_object_id]:k:[signature_object_key]`.
- a boolean that indicates if the commit is a snapshot. (in this case, only one commit is present in the HEADs)
### SignatureRequest
@ -867,7 +867,7 @@ AppRequestV0 {
nuri: NuriV0 {
target: NuriTargetV0::Repo(repo_id),
overlay: Some(overlay_id),
branch: //not implemented. defaults to main branch
branch: //not implemented. Defaults to main branch
... // all the rest empty
},
payload: None
@ -894,7 +894,7 @@ AppRequestV0 {
nuri: NuriV0 {
target: NuriTargetV0::Repo(repo_id),
overlay: Some(overlay_id),
branch: //not implemented. defaults to main branch
branch: //not implemented. Defaults to main branch
The Client Protocol is used by the Verifier in order to contact the Broker of the User.
It maintain this connection throughout the session that was opened by the User (by opening the wallet in the app, by example).
It maintains this connection throughout the session that was opened by the User (by opening the wallet in the app, by example).
From this connection, the Verifier gets all the pushed updates (called Events), after it subscribed to some topics.
The verifier also sends the updates that it wants to publish, in the form of an Event to the Pub/Sub Topic, and the Broker deals with forwarding this Event to all the other devices and users that have subscribed to this topic.
The communication on the Client Protocol are using a WebSocket, encrypted from within with the Noise Protocol.
The communication on the Client Protocol is using a WebSocket, encrypted from within with the Noise Protocol.
In addition, all the Events, that are send and received with this protocol, are also encrypted end-to-end.
In addition, all the Events, that are sent and received with this protocol, are also encrypted end-to-end.
For now, the Verifier only connects to one Broker, but for redundancy and failsafe purposes, it will be possible in the future that it tries to connect to several Brokers.
But one rule should always be followed: for any given Overlay, a User can only participate in this Overlay from one and only one Broker at the same time.
Let's dive into the format of the messages and actions/commands that can be exchange on the Client Protocol
Let's dive into the format of the messages and actions/commands that can be exchanged on the Client Protocol.
The initiation of the connection is common to all protocols, and involves some Noise handshake. It isn't detailed here, please refer to the code for now. We will provide more documentation on that part later on.
@ -30,7 +30,7 @@ For a reference of the common types, please refer to the [Repo format documentat
### ClientMessage
All the subsequent message sent and receive on this protocol, are encapsulated inside a `ClientMessage`.
All the subsequent messages sent and receive on this protocol, are encapsulated inside a `ClientMessage`.
The `ClientRequestV0.id` is set by the requester in an incremental way. Request IDs must be unique by session. They should start from 1 after every start of a new session. This ID is present in the response, in order to match requests and responses.
- 1 means PartialContent (the response is a stream. each element in the stream will have this result code)
- 1 means PartialContent (the response is a stream. Each element in the stream will have this result code)
- 2 means EndOfStream (that's the last response in the stream)
- 3 means False
- 4 and above are errors. for the list, see `ng-repo/src/errors.rs` starting at line 265.
- 4 and above are errors. For the list, see `ng-repo/src/errors.rs` starting at line 265.
When an error occurs (result >= 3), the content is of the type ClientResponseContentV0::EmptyResponse
@ -167,12 +167,10 @@ Request a Commit by ID
Replied with a stream of `Block`s.
`commit_header_key` of the replied blocks is always set to None when request is made on OuterOverlay of protected or Group overlays.
`commit_header_key` of the replied blocks is always set to None when request is made on OuterOverlay of Protected or Group overlays.
The difference with BlocksGet is that the Broker will try to return all the commit blocks as they were sent in the Pub/Sub Event, if it has it.
This will help in having all the blocks (including the header and body blocks) in one ClientProtocol message, while a BlocksGet would inevitably return only the blocks of the ObjectContent,
and not the header nor the body. And the load() would fail with CommitLoadError::MissingBlocks. That's what happens when the Commit is not present in the pubsub,
and we need to default to using BlocksGet instead.
The difference with BlocksGet is that the Broker will try to return all the commit blocks as they were sent in the Pub/Sub Event, if it has the event.
This will help in receiving all the blocks (including the header and body blocks) in one ClientProtocol message, while a BlocksGet would inevitably return only the blocks of the ObjectContent, and not the header nor the body. And the load() would fail with CommitLoadError::MissingBlocks. That's what happens when the Commit is not present in the pubsub, and then we need to default to using BlocksGet instead.
#### Request
@ -194,11 +192,11 @@ struct CommitGetV0 {
### RepoPinStatus
Request the status of pinning for a repo on the broker (for the current user's session).
Request the pinning status for a repo on the broker (for the current user's session).
Returns an error code if not pinned, otherwise returns a RepoPinStatusV0.
The overlay entered in ClientMessage is important. if it is the outer, only outer pinning will be checked.
/// if it is the inner overlay, only the inner pinning will be checked.
The overlay entered in ClientMessage is important. Id it is the outer, only outer pinning will be checked.
If it is the inner overlay, only the inner pinning will be checked.
#### Request
@ -318,11 +316,11 @@ enum OverlayAccess {
- PinRepoV0.overlay_root_topic : Root topic of the overlay, used to listen to overlay refreshes. Only set for inner overlays (RW or WO) overlays. Not implemented yet
- PinRepoV0.expose_outer : only possible for RW overlays. not allowed for private or dialog overlay. not implemented yet
- PinRepoV0.expose_outer : only possible for RW overlays. Not allowed for private or dialog overlay. Not implemented yet
- PinRepoV0.peers : Broker peers to connect to in order to join the overlay. If the repo has previously been opened (during the same session) or if it is a private overlay, then peers info can be omitted. If there are no known peers in the overlay yet, vector is left empty (creation of a store, or repo in a store that is owned by user).
- PinRepoV0.max_peer_count : Maximum number of peers to connect to for this overlay (only valid for an inner (RW/WO) overlay). not implemented yet
- PinRepoV0.max_peer_count : Maximum number of peers to connect to for this overlay (only valid for an inner (RW/WO) overlay). Not implemented yet
- PinRepoV0.ro_topics : list of topics that should be subscribed to. If the repo has previously been opened (during the same session) and the list of RO topics does not need to be modified, then ro_topics info can be omitted
@ -367,7 +365,7 @@ struct TopicSubV0 {
#### Response
A `TopicSubRes`. see above in [RepoPinStatus](#repopinstatus) for more details.
A `TopicSubRes`. See above in [RepoPinStatus](#repopinstatus) for more details.
### TopicSyncReq
@ -420,7 +418,7 @@ enum TopicSyncResV0 {
Request to know if some blocks are present locally on the responder
used by a Client before publishing an event with FILES, to know what to push, and save bandwidth if the blocks are already present on the Broker (content deduplication). commits without FILES cannot be deduplicated because they are unique, due to their unique position in the DAG, and the embedded BranchId.
used by a Client before publishing an event with FILES, to know what to push, and save bandwidth if the blocks are already present on the Broker (content deduplication). Commits without FILES cannot be deduplicated because they are unique, due to their unique position in the DAG, and the embedded BranchId.
@ -48,7 +48,7 @@ The 2 Javascript libraries do not have a User Storage so they only support in-me
As the feature of the “User Storage for Web” will take some time to be coded, so we offer another way to solve the problem of volatile materialized state in JS.
There is also the idea of having a full fledged Verifier running inside nodeJS. this would use the NAPI-RS system which compiles the Rust code of the verifier, together with the RocksDb code, and make it a binary library compatible with nodeJS that would run inside the nodeJS process. This also will take some time to be coded.
There is also the idea of having a full fledged Verifier running inside nodeJS. This would use the NAPI-RS system which compiles the Rust code of the verifier, together with the RocksDb code, and make it a binary library compatible with nodeJS that would run inside the nodeJS process. This also will take some time to be coded.
Instead for both cases (JS in web and in nodeJS) we offer the App API that connects to a remote Verifier.
@ -60,13 +60,13 @@ Usually `ngd` only acts as a broker, but it can be configured and used as a Veri
in which use cases is it useful ?
- when the end-user doesn’t have a supported platform where to install the native app. by example, a workstation running OpenBSD or FreeBSD, doesn't have a native app to download (and cannot be compiled neither as Tauri doesn't support such platform). In this case, the end-user has to launch a local ngd, and open the webapp from their browser (http://localhost:1440). The verifier will run remotely, inside ngd (that isn't very far. it is on the same machine). Because it is on localhost or in a private LAN, we do allow the webapp to be served on http (without TLS) and the websocket is also working well without TLS. But this doesn't work anymore if the public IP of the ngd server is used.
- when the end-user doesn’t have a supported platform where to install the native app. By example, a workstation running OpenBSD or FreeBSD, doesn't have a native app to download (and cannot be compiled neither as Tauri doesn't support such platform). In this case, the end-user has to launch a local ngd, and open the webapp from their browser (http://localhost:1440). The verifier will run remotely, inside ngd (that isn't very far, it is on the same machine). Because it is on localhost or in a private LAN, we do allow the webapp to be served on http (without TLS) and the websocket is also working well without TLS. But this doesn't work anymore if the public IP of the ngd server is used.
- when a nodeJS service needs access to the documents and does not want to use the in-memory Verifier, because it needs quick access (like a headless CMS, Astro, an AI service like jan.ai, or a SPARQL REST endpoint, an LDP endpoint, etc..) then in this case, an ngd instance has to run in the same machine as the nodeJS process, or in the same LAN network (Docker network by example).
- in the headless mode, when a server is using ngd as a quadstore/document store and the full credentials of the user identity has been delegated to that server. This is the case for ActivityPods, by example.
- on the SaaS/cloud of NextGraph, we run some ngd brokers that normally would not have any verifier. But in some cases, at the request of the end-user, we can run some verifiers that have limited access to some documents or stores of the user. if they want to serve their data as REST/HTTP endpoints, by example. The end-user will have to grant access about those resources to this remote verifier, by providing their DID capabilities. A Verifier can see in clear all the data that it manipulates, so obviously, users have to be careful where they run a Verifier, and to whom they give the capabilities.
- on the SaaS/cloud of NextGraph, we run some ngd brokers that normally would not have any verifier. But in some cases, at the request of the end-user, we can run some verifiers that have limited access to some documents or stores of the user. If they want to serve their data as REST/HTTP endpoints, by example. The end-user will have to grant access about those resources to this remote verifier, by providing their DID capabilities. A Verifier can see in clear all the data that it manipulates, so obviously, users have to be careful where they run a Verifier, and to whom they give the capabilities.
What is important to understand is that the Verifier needs to run in a trusted environment because it holds the ReadCaps of the documents it is going to open, and in some cases, it even holds the full credentials of the User Identity and has access to the whole set of documents in all stores of the user.
@ -97,11 +97,11 @@ The Verifier talks to the Broker with the ClientProtocol, and receives the encry
Then, it exposes the AppProtocol to the Application level, this is what is used to access and modify the data.
Sometimes the Verifier and the Broker are on the same machine, in the same process, so they use the LocalTransport which is not even using the network interface. That’s the beauty of the code of NextGraph. it has been thought from the beginning with many use cases in mind.
Sometimes the Verifier and the Broker are on the same machine, in the same process, so they use the LocalTransport which is not even using the network interface. That’s the beauty of the code of NextGraph. It has been thought from the beginning with many use cases in mind.
Sometimes the Verifier and the App are in the same process, sometimes they need a websocket between them. But all of this are implementation details. For the developers, the same API is available everywhere, in nodeJS, in front-end Javascript, in Rust, and similarly, as commands in the CLI, regardless of where the Verifier and the Broker are actually located.
In some cases, a broker (ngd) will run. let's say on localhost or within a LAN network, and will not be directly connected to the core network. This can happen in the following schema. This is called a Server Broker, and it doesn’t join the core network. Instead, it needs to establish a connection to a CoreBroker that will join the core network on its behalf. It will use the ClientProtocol for that, in a special way called “Forwarding", as it will forward all ClientProtocol request coming from the Verifier(s), to another broker called the CoreBroker. It will keep local copies of the events, and manage a local table of pub/sub subscriptions, but will not join overlays by itself. This will be delegated to the CoreBroker(s) it connects to.
In some cases, a broker (ngd) will run. Let's say on localhost or within a LAN network, and will not be directly connected to the core network. This can happen in the following schema. This is called a Server Broker, and it doesn’t join the core network. Instead, it needs to establish a connection to a CoreBroker that will join the core network on its behalf. It will use the ClientProtocol for that, in a special way called “Forwarding", as it will forward all ClientProtocol request coming from the Verifier(s), to another broker called the CoreBroker. It will keep local copies of the events, and manage a local table of pub/sub subscriptions, but will not join overlays by itself. This will be delegated to the CoreBroker(s) it connects to.
This Forwarding Client Protocol is not coded yet (but it is just an add-on to the ClientProtocol).
@ -16,13 +16,13 @@ We have designed a new wallet specially for our needs, and we have innovative fe
A wallet should be unique to a physical human being. We cannot strictly enforce that, but we would like this rule to be respected, in order to simplify a bit the use cases. So we recommend each person to create only one Wallet for themselves.
Inside a Wallet, many Identities can coexist. A physical individual can create several identities for themselves inside the same wallet. By default, when the Wallet is created, a Personal identity is also created within it. That's the default identity. Then, the individual can add more identities, that will be untraceable back to the wallet, meaning that each identity is unique and has no link to the wallet or to other identities within it. When interacting with other Users, if distinct Identities are used, there will be no way to correlate those distinct Identities, even if they are stored in the same Wallet. By example, one can use the default Personal identity for friends and family, but then create another identity for their professional interactions. it will be impossible for their coworker or boss to find out what is their personal identity, and vis versa. the number of additional identities is unlimited. They are stored in the wallet and just grows the size of the wallet. This feature guarantees total anonymity and separation of Identities. In NextGraph terminology, an Identity is the same as a User.
Inside a Wallet, many Identities can coexist. A physical individual can create several identities for themselves inside the same wallet. By default, when the Wallet is created, a Personal identity is also created within it. That's the default identity. Then, the individual can add more identities, that will be untraceable back to the wallet, meaning that each identity is unique and has no link to the wallet or to other identities within it. When interacting with other Users, if distinct Identities are used, there will be no way to correlate those distinct Identities, even if they are stored in the same Wallet. By example, one can use the default Personal identity for friends and family, but then create another identity for their professional interactions. It will be impossible for their coworker or boss to find out about their personal identity, and vice versa. The number of additional identities is unlimited. They are stored in the wallet and just grows the size of the wallet. This feature guarantees total anonymity and separation of Identities. In NextGraph terminology, an Identity is the same as a User.
Then we also have the concept of an Organization.
An Organization is another type of Identity. It has the same content as a Personal Identity. But in addition, it has member Users, which are Individual Identities that have been associated with the Organization, in a similar way as with email addresses at individual@organization .
An Organization can also have sub-organization, in a hierarchical manner. as in organization/sub-organization .
An Organization can also have sub-organization, in a hierarchical manner, as in `organization/sub-organization`.
An Organization has Owners, and the ownership can be transferred to other Identities, while this is not the case for Individual Identities that cannot be transferred.
@ -46,7 +46,7 @@ Once the wallet has arrived on a new device, everything works the same in the ne
In order to protect your wallet from unauthorized access, we have decided not to use a password, because that would be too risky. We know very well that users do not choose secure password by themselves, because they need to remember such password, so it has to be simple.
The other opposite behaviour is to create a very secure password with a password generator, by example, and then store this very complex password in a password/keys manager. This is another problem, as it just transfers the security of the whole system to that "password manager" that we know have been found insecure so many times. the question of the transfer of such complex password from one device to another is another problem... and we wouldn't have solved anything if we were to use passwords.
The other opposite behaviour is to create a very secure password with a password generator, by example, and then store this very complex password in a password/keys manager. This is another problem, as it just transfers the security of the whole system to that "password manager" that we know have been found insecure so many times. The question of the transfer of such complex password from one device to another is another problem... And we wouldn't have solved anything if we were to use passwords.
So.. here comes the Pazzle.
@ -62,7 +62,7 @@ There is also a mnemonic "passphrase" that is an alternative way to login into y
We propose this option for those we have special needs (like programmers that need to enter in the wallet very quickly).
In general, we encourage you to use the pazzle instead. in the future, we will also implement the option to login with physical dongle/key that has been paired with the wallet.
In general, we encourage you to use the pazzle instead. In the future, we will also implement the option to login with physical dongle/key that has been paired with the wallet.
The mnemonic can also be seen as a recovery passphrase. But be very careful where you save it. Anybody that finds your mnemonic or pazzle, and also has a hold on the wallet file or on a device that has the wallet already imported, can surely enter your account and read/write all your data.
Niko PLP
Laurin Weger