docs-site

Documentation site of NextGraph.org https://docs.nextgraph.org

23 KiB

Raw Blame History

title	description	layout
App Protocol	Specification of the App Protocol	../../../layouts/MainLayout.astro

All our protocols and formats use the binary codec called BARE.

The App Protocol lets the Application talk with the Verifier.

This protocol exchanges content that isn't encrypted.

In the native apps, this protocol runs between the front-end part of the App and the back-end part that is running inside Tauri from Rust compiled into the app binary. Both sides of the protocol are in the same process, but there is an interface between them provided by Tauri (which is made of JSON objects), but this is transparent to the App developer.

In the web-app, the same happens, but the "back-end" part runs inside the same JS context, and is some WASM code compiled from Rust. The passing of messages doesn't involve JSON, as JS POJOs are directly exchanged at the interface.

If the App opens a session with a remote Verifier, the protocol is ran inside a WebSocket, with Noise encryption.

This protocol is available directly from the JS and Rust APIs and you do not need to implement it again or send messages manually. New bindings are always welcomed, for any contributor who wants to add them.

All the protocol messages are embedded inside an AppRequest message, and are replied by the Verifier with an AppResponse.

Some Requests will respond with a stream of AppResponses, while others will respond with only one.

Streamed responses :

Fetch(Subscribe)
FileGet

A new session needs to be opened with the LocalBroker methods, not documented here (it will be documented in the APIs reference). Once the session is opened, the session_id needs to be passed in every request.

For a reference of the common types, please refer to the Repo format documentation

AppRequest

Request

enum AppRequest {
    V0(AppRequestV0),
}

struct AppRequestV0 {
    command: AppRequestCommandV0,

    nuri: NuriV0,

    payload: Option<AppRequestPayload>,

    session_id: u64,
}

enum AppRequestCommandV0 {
    Fetch(AppFetchContentV0),
    Pin,
    UnPin,
    Delete,
    Create,
    FileGet,
    FilePut,
}

enum AppFetchContentV0 {
    Get, // without subscribing
    Subscribe,
    Update,
    ReadQuery,
    WriteQuery,
    RdfDump,
    History,
    SignatureStatus,
    SignatureRequest,
    SignedSnapshotRequest,
}

enum AppRequestPayload {
    V0(AppRequestPayloadV0),
}

enum AppRequestPayloadV0 {
    Create(DocCreate),
    Query(DocQuery),
    Update(DocUpdate),
    AddFile(DocAddFile),
    Delete(DocDelete),
    SmallFilePut(SmallFile),
    /// content_type (IANA media type)
    RandomAccessFilePut(String),
    /// an empty Vec ends the upload
    RandomAccessFilePutChunk((u32, Vec<u8>)),
}

struct NuriV0 {

    identity: Option<UserId>,
    target: NuriTargetV0,
    entire_store: bool,

    objects: Vec<ObjectRef>,
    signature: Option<ObjectRef>,

    branch: Option<TargetBranchV0>,
    overlay: Option<OverlayLink>,

    access: Vec<NgAccessV0>,
    topic: Option<TopicId>,
    locator: Option<Locator>,
}

enum NuriTargetV0 {
    /// targets the whole DataSet of the user
    UserSite,

    PublicStore,
    ProtectedStore,
    PrivateStore,
    AllDialogs,
    /// shortname of a Dialog
    Dialog(String),
    AllGroups,
    /// shortname of a Group
    Group(String),

    Repo(RepoId),

    None,
}

enum TargetBranchV0 {
    Chat,
    Stream,
    Comments,
    BackLinks,
    Context,
    BranchId(BranchId),
    /// named branch or commit
    Named(String),
    Commits(Vec<ObjectId>),
}

enum OverlayLink {
    Outer(Digest),
    InnerLink(InnerOverlayLink),
    Inner(Digest),
    Inherit,
    Public(PubKey),
    Global,
}

struct InnerOverlayLink {
    /// overlay public key ID
    id: StoreOverlay,
    store_overlay_readcap: ReadCap,
}

enum NgAccessV0 {
    ReadCap(ReadCap),
    Token(Digest),
    ExtRequest(Vec<u8>),
    Key(BlockKey),
    Inbox(PubKey),
}

enum Locator {
    V0(LocatorV0),
}

type LocatorV0 = Vec<BrokerServer>;

struct BrokerServer {
    content: BrokerServerContentV0,

    /// peerId of the server
    peer_id: PubKey,

    /// optional signature over content by peer_id
    sig: Option<Sig>,
}

struct BrokerServerContentV0 {
    servers: Vec<BrokerServerTypeV0>,
    version: u32,
}

enum BrokerServerTypeV0 {
    /// optional port number, defaults to 1440
    Localhost(u16),
    BoxPrivate(Vec<BindAddress>),
    Public(Vec<BindAddress>),
    BoxPublicDyn(Vec<BindAddress>),
    Domain(String),
}

NuriV0.identity : None for personal identity
NuriV0.entire_store : If it is a store, will include all the docs belonging to the store. not used otherwise
NuriV0.objects : used only for FileGet.
NuriV0.branch : if None, the main branch is chosen
InnerOverlayLink.store_overlay_readcap : The store has a special branch called Overlay that is used to manage access to the InnerOverlay. Only the ReadCapSecret is needed to access the InnerOverlay. The full readcap of this branch is needed in order to subscribe to the topic and decrypt the events, and hence be able to subscribe to refreshes of the InnerOverlay. The branchId can be found in the branch Definition. It can be useful to subscribe to this topic if the user is a member of the store's repo, so it will be notified of BranchCapRefresh on the overlay. To the contrary, if the user is an external user to the store, they won't be able to subscribe and they will loose access to the InnerOverlay after a BranchCapRefresh of the overlay branch of the store.
BrokerServerTypeV0::Domain : accepts an optional trailing ":port" number

Response

enum AppResponse {
    V0(AppResponseV0),
}

enum AppResponseV0 {
    SessionStart(AppSessionStartResponse),
    TabInfo(AppTabInfo),
    State(AppState),
    Patch(AppPatch),
    History(AppHistory),
    SignatureStatus(Vec<(String, Option<String>, bool)>),
    Text(String),
    FileUploading(u32),
    FileUploaded(ObjectRef),
    FileBinary(Vec<u8>),
    FileMeta(FileMetaV0),
    QueryResult(Vec<u8>),
    Graph(Vec<u8>),
    Ok,
    True,
    False,
    Error(String),
    EndOfStream,
    Nuri(String),
}

enum AppSessionStartResponse {
    V0(AppSessionStartResponseV0),
}

struct AppSessionStartResponseV0 {
    private_store: RepoId,
    protected_store: RepoId,
    public_store: RepoId,
}

DocCreate

Creates a new Document.

Request

// example AppRequestV0
AppRequestV0 {
    ...,
    command : AppRequestCommandV0::Create,
    nuri: NuriV0::new_empty(),
    payload: Some(AppRequestPayload::V0(AppRequestPayloadV0::Create(
        DocCreate {
            ...
        },
    ))),
}

struct DocCreate {
    store: StoreRepo,
    class: BranchCrdt,
    destination: DocCreateDestination,
}

enum DocCreateDestination {
    Store,
    Stream,
    MagicCarpet,
}

Response

replies with an AppResponseV0::Nuri(string) containing the string representation of the Document's Nuri, of the form did:ng:o:[repo_id]:v:[overlay_id].

DocUpdate

Updates the graph or discrete nature of the Document, or both.

Replied with AppResponseV0::Ok.

If set, the NuriV0.branch should be of the variant BranchId(_). If None, the main branch will be used.

Request

// example AppRequestV0
AppRequestV0 {
    ...,
    command : AppRequestCommandV0::Fetch(AppFetchContentV0::Update),
    nuri: NuriV0 {
        target: NuriTargetV0::Repo(repo_id),
        overlay: Some(overlay_id),
        ... // all the rest empty
    },
    payload: Some(AppRequestPayload::V0(AppRequestPayloadV0::Update(
        DocUpdate {
            ...
        },
    ))),
}

struct DocUpdate {
    heads: Vec<ObjectId>,
    graph: Option<GraphUpdate>,
    discrete: Option<DiscreteUpdate>,
}

struct GraphUpdate {
    // serialization of Vec<Triple>
    inserts: Vec<u8>,
    // serialization of Vec<Triple>
    removes: Vec<u8>,
}

enum DiscreteUpdate {
    /// A yrs::Update
    YMap(Vec<u8>),
    YArray(Vec<u8>),
    YXml(Vec<u8>),
    YText(Vec<u8>),
    /// An automerge::Change.raw_bytes()
    Automerge(Vec<u8>),
}

GraphUpdate is not implemented for now.

WriteQuery

A SPARQL Update query. Can span multiple Documents (by indicating a GRAPH <Nuri> or WITH <Nuri>, and optional USING <Nuri> with a Nuri of the form did:ng:o:v).

The AppRequestV0.nuri is mandatory and will represent the default graph.

Replied with a AppResponseV0::Ok.

The NuriV0.target cannot be a UserSite, AllDialogs nor AllGroups.

The NuriV0.branch cannot be a Commits(_).

// example AppRequestV0
AppRequestV0 {
    ...,
    command : AppRequestCommandV0::Fetch(AppFetchContentV0::WriteQuery),
    nuri: NuriV0 {
        target: NuriTargetV0::Repo(repo_id),
        overlay: Some(overlay_id),
        ... // all the rest empty
    },
    payload: Some(AppRequestPayload::V0(AppRequestPayloadV0::Query(
        DocQuery::V0 {
            ...
        },
    ))),
}

enum DocQuery {
    V0 {
        sparql: String,
        base: Option<String>,
    },
}

DocQuery::V0.base : an optional base to resolve all your relative URIs of resources in the SPARQL Update query.
DocQuery::V0.sparql : the text of your SPARQL Update

FilePut

Uploads a binary file into the repository (Document). The API is composed of several calls that should be made sequentially.

The first call must be an AppRequestPayloadV0::RandomAccessFilePut that will return an upload_id.

Immediately followed by one or more AppRequestPayloadV0::RandomAccessFilePutChunk.

And eventually finished with one call to AppRequestPayloadV0::AddFile.

first Request : RandomAccessFilePut

// example AppRequestV0
AppRequestV0 {
    ...,
    command : AppRequestCommandV0::FilePut,
    nuri: NuriV0 {
        target: NuriTargetV0::Repo(repo_id),
        overlay: Some(overlay_id),
        ... // all the rest empty
    },
    payload: Some(AppRequestPayload::V0(AppRequestPayloadV0::RandomAccessFilePut(
        content_type // a string representing an IANA media type
    ))),
}

first Response

AppResponseV0::FileUploading(upload_id) // a u32

one or more chunk put Request(s) : RandomAccessFilePutChunk

upload your file in chunks of maximum 1 048 564 bytes, for best efficiency.

Repeat this call until you finished upload all your file.

Add another extra call at the end, with a size of zero. This will indicate that you are done with uploading chunks.

Replied with a AppResponseV0::Ok for each chunk that is non-empty.

The last call with an empty chunk is replied with an AppResponseV0::FileUploaded(reference) containing an ObjectRef to the uploaded file, that you should use in the next call AppRequestPayloadV0::AddFile.

// example AppRequestV0
AppRequestV0 {
    ...,
    command : AppRequestCommandV0::FilePut,
    nuri: NuriV0::new_empty(),
    payload: Some(AppRequestPayload::V0(
        AppRequestPayloadV0::RandomAccessFilePutChunk(
            (upload_id, chunk) // chunk is a Vec<u8>
        )
    )),
}

nuri : can be omitted in those calls, unlike the 2 other type of calls

last Request : AddFile

Finally you make an AppRequestPayloadV0::AddFile call that will attach the Object to your document's branch.

// example AppRequestV0
AppRequestV0 {
    ...,
    command : AppRequestCommandV0::FilePut,
    nuri: NuriV0 {
        target: NuriTargetV0::Repo(repo_id),
        overlay: Some(overlay_id),
        branch: // can be omitted (defaults to main branch)
        ... // all the rest empty
    },
    payload: Some(AppRequestPayload::V0(
        AppRequestPayloadV0::AddFile(
            DocAddFile {
                ...
            }
        )
    )),
}

struct DocAddFile {
    filename: Option<String>,
    object: ObjectRef,
}

DocAddFile.object : must be the reference you obtained from the last call to RandomAccessFilePutChunk.
DocAddFile.filename : an optional filename. usually it is the original filename on the filesystem when selecting the binary file to upload.

final Response

Finally, you get an AppResponseV0::Ok if everything went well.

FileGet

Downloads a binary file

The request nuri needs to have both a target and one, and only one objects.

Replied with a stream of AppResponseV0.

Request

// example AppRequestV0
AppRequestV0 {
    ...,
    command : AppRequestCommandV0::FileGet,
    nuri: NuriV0 {
        target: NuriTargetV0::Repo(repo_id),
        overlay: Some(overlay_id),
        branch: // can be omitted (defaults to main branch)
        objects: [object_ref], // the requested object/file reference
        ... // all the rest empty
    },
    payload: None,
}

first Response : FileMeta

The first response in the stream is always an AppResponseV0::FileMeta(FileMetaV0).

struct FileMetaV0 {
    content_type: String,
    size: u64,
}

more Responses : FileBinary

Then a series of AppResponseV0::FileBinary(chunk) containing all the chunks (Vec<u8>) of the file, are send in the streamed response.

final response : EndOfStream

And eventually, once all the data has been transferred, an AppResponseV0::EndOfStream is sent in the stream.

Subscribes to all the events/commits of a branch.

The response is streamed and long-lived, until cancelled.

The first response contains some meta-data about the branch called TabInfo. It is used mostly by the App to display the document header and instantiate the document's viewer.

Then a second response contains the materialized state of the document as it is known by the local Verifier at the moment of the subscribe request.

Later on, new updates are added to the stream as they arrive on the local replica (including from the changes made locally by the user).

Request

// example AppRequestV0
AppRequestV0 {
    ...,
    command : AppRequestCommandV0::Fetch(AppFetchContentV0::Subscribe),
    nuri: NuriV0 {
        target: NuriTargetV0::Repo(repo_id),
        overlay: Some(overlay_id),
        branch: // can be omitted (defaults to main branch)
        ... // all the rest empty
    },
    payload: None,
}

first Response : AppTabInfo

The first response is a AppResponseV0::TabInfo(AppTabInfo).

struct AppTabInfo {
    branch: Option<AppTabBranchInfo>,
    doc: Option<AppTabDocInfo>,
    store: Option<AppTabStoreInfo>,
}

struct AppTabBranchInfo {
    id: Option<String>,
    readcap: Option<String>,
    comment_branch: Option<String>, // not implemented
    class: Option<String>,
}

struct AppTabDocInfo {
    nuri: Option<String>,
    is_store: Option<bool>,
    is_member: Option<String>,
    title: Option<String>,
    icon: Option<String>,
    description: Option<String>,
    authors: Option<Vec<String>>, // not implemented
    inbox: Option<String>, // not implemented
    can_edit: Option<bool>,
}

struct AppTabStoreInfo {
    repo: Option<StoreRepo>,
    overlay: Option<String>,
    has_outer: Option<String>, // not implemented
    store_type: Option<String>,
    readcap: Option<String>,
    is_member: Option<String>,
    inner: Option<String>,
    title: Option<String>,
    icon: Option<String>,
    description: Option<String>,
}

second Response : AppState

The second response is a AppResponseV0::State(AppState).

struct AppState {
    pub heads: Vec<ObjectId>,
    pub head_keys: Vec<ObjectKey>,
    pub graph: Option<GraphState>,
    pub discrete: Option<DiscreteState>,
    pub files: Vec<FileName>,
}

struct GraphState {
    // serialization of Vec<Triple>
    pub triples: Vec<u8>,
}

enum DiscreteState {
    /// A yrs::Update
    YMap(Vec<u8>),
    YArray(Vec<u8>),
    YXml(Vec<u8>),
    YText(Vec<u8>),
    // the output of Automerge::save()
    Automerge(Vec<u8>),
}

struct FileName {
    pub name: Option<String>,
    pub reference: ObjectRef,
    pub nuri: String,
}

followed by AppPatch

Then on every update done on the branch, a new AppResponseV0::Patch(AppPatch) is sent in the stream.

struct AppPatch {
    pub commit_id: String,
    pub commit_info: CommitInfoJs,
    // or graph, or discrete, or both, or other.
    pub graph: Option<GraphPatch>,
    pub discrete: Option<DiscretePatch>,
    pub other: Option<OtherPatch>,
}

struct CommitInfoJs {
    pub past: Vec<String>,
    pub key: String,
    pub signature: Option<String>,
    pub author: String,
    pub timestamp: String,
    pub final_consistency: bool,
    pub commit_type: CommitType,
    pub branch: Option<String>,
    pub x: u32,
    pub y: u32,
}

enum CommitType {
    TransactionGraph,
    TransactionDiscrete,
    TransactionBoth,
    FileAdd,
    FileRemove,
    Snapshot,
    Compact,
    AsyncSignature,
    SyncSignature,
    Branch,
    UpdateBranch,
    BranchCapRefresh,
    CapRefreshed,
    Other,
}

struct GraphPatch {
    // serialization of Vec<Triple>
    pub inserts: Vec<u8>,
    // serialization of Vec<Triple>
    pub removes: Vec<u8>,
}

enum DiscretePatch {
    /// A yrs::Update
    YMap(Vec<u8>),
    YArray(Vec<u8>),
    YXml(Vec<u8>),
    YText(Vec<u8>),
    /// An automerge::Change.raw_bytes() or a concatenation of several of them.
    Automerge(Vec<u8>),
}

enum OtherPatch {
    FileAdd(FileName),
    FileRemove(ObjectId),
    AsyncSignature((String, Vec<String>)),
    Snapshot(ObjectRef),
    Compact(ObjectRef),
    Other,
}

ReadQuery

A SPARQL query (read-only).

Request

// example AppRequestV0
AppRequestV0 {
    ...,
    command : AppRequestCommandV0::Fetch(AppFetchContentV0::ReadQuery),
    nuri: NuriV0 {
        target: NuriTargetV0::Repo(repo_id),
        overlay: Some(overlay_id),
        branch: // not implemented yet
        ... // all the rest empty
    },
    payload: Some(AppRequestPayload::V0(
        AppRequestPayloadV0::Query(
            DocQuery::V0 {
                ...
            }
        )
    )),
}

enum DocQuery {
    V0 {
        sparql: String,
        base: Option<String>,
    },
}

AppRequestV0.nuri.target : represents the default graph. can be NuriV0::UserSite or NuriV0::None and in those cases, the union graph of all the graphs is used as default graph.
DocQuery::V0.base : an optional base to resolve all your relative URIs of resources in the SPARQL Query.
DocQuery::V0.sparql : the text of your SPARQL Query

Response

Depending on the type of query, the response differs.

for SELECT queries: an AppResponseV0::QueryResult(buffer) where buffer is a Vec<u8> containing a UTF-8 serialization of a JSON string representing a JSON Sparql Query Result. see SPARQL Query Results JSON Format.
for CONSTRUCT an AppResponseV0::Graph(buffer) where buffer is a Vec<u8> containing a BARE serialization of a Vec<Triple>.
for ASK queries: AppResponseV0::True or AppResponseV0::False

RdfDump

Gets the full dump of all the quads contained in the UserSite (whole local DataSet of the User).

Request

// example AppRequestV0
AppRequestV0 {
    ...,
    command : AppRequestCommandV0::Fetch(AppFetchContentV0::RdfDump),
    nuri: NuriV0::new_empty()
    payload: None,
}

Response

AppResponseV0::Text(string)

string being a Turtle output of all the quads.

History

Fetches the current (HEADs) history of commits for this branch.

Request

// example AppRequestV0
AppRequestV0 {
    ...,
    command : AppRequestCommandV0::Fetch(AppFetchContentV0::History),
    nuri: NuriV0 {
        target: NuriTargetV0::Repo(repo_id),
        overlay: Some(overlay_id),
        branch: //not implemented. defaults to main branch
        ... // all the rest empty
    },
    payload: None
}

Response

The response is a AppResponseV0::History(AppHistory)

struct AppHistory {
    pub history: Vec<(ObjectId, CommitInfo)>,
    pub swimlane_state: Vec<Option<ObjectId>>,
}

struct CommitInfo {
    pub past: Vec<ObjectId>,
    pub key: ObjectKey,
    pub signature: Option<ObjectRef>,
    pub author: String,
    pub timestamp: Timestamp,
    pub final_consistency: bool,
    pub commit_type: CommitType,
    pub branch: Option<ObjectId>,
    pub x: u32,
    pub y: u32,
}

AppHistory.history : in order, with the newest commits first. The first part of the tuple (ObjectId) is the CommitID.
AppHistory.swimlane_state : can be discarded. it is only used by our GUI representation of the history in the Apps. Same with the x and y values in CommitInfo.

SignatureStatus

Fetches the Signature status of the branch, at the HEADs.

Request

// example AppRequestV0
AppRequestV0 {
    ...,
    command : AppRequestCommandV0::Fetch(AppFetchContentV0::SignatureStatus),
    nuri: NuriV0 {
        target: NuriTargetV0::Repo(repo_id),
        overlay: Some(overlay_id),
        branch: //not implemented. defaults to main branch
        ... // all the rest empty
    },
    payload: None
}

Response

The response is a AppResponseV0::SignatureStatus(Vec<(String, Option<String>, bool)>)

Which is a list of commits at the HEAD. Each commit is represented by a tuple that contains :

the commit ID printed as a string (44 characters)
an optional string that is present only if the commit is a signature. in this case, the string represents a list of the signed commits c:[commit_id]:k:[commit_key] joined with : (at least one commit is present), followed by partial Nuri for the signature object :s:[signature_object_id]:k:[signature_object_key].
a boolean that indicates if the commit is a snapshot. (in this case, only one commit is present in the HEADs)

SignatureRequest

Requests that the HEADs commit(s) be asynchronously signed.

Request

// example AppRequestV0
AppRequestV0 {
    ...,
    command : AppRequestCommandV0::Fetch(AppFetchContentV0::SignatureRequest),
    nuri: NuriV0 {
        target: NuriTargetV0::Repo(repo_id),
        overlay: Some(overlay_id),
        branch: //not implemented. defaults to main branch
        ... // all the rest empty
    },
    payload: None
}

Response

If the signature is immediately available, the response is AppResponseV0::True. The new HEAD containing the signature can be immediately fetched with a call to SignatureStatus.

If the signature is not immediately available (because the quorum needs to sign it and it can take some time), the response is AppResponseV0::False. A notification will be sent to the user once the signature is ready (not implemented yet).

SignedSnapshotRequest

Requests that a snapshot be taken from the current materialized state at the HEADs, and that this new commit containing the snapshot, be signed asynchronously.

Request

// example AppRequestV0
AppRequestV0 {
    ...,
    command : AppRequestCommandV0::Fetch(AppFetchContentV0::SignedSnapshotRequest),
    nuri: NuriV0 {
        target: NuriTargetV0::Repo(repo_id),
        overlay: Some(overlay_id),
        branch: //not implemented. defaults to main branch
        ... // all the rest empty
    },
    payload: None
}

Response

If the signature is immediately available, the response is AppResponseV0::True. The newly created snapshot and its signature can be immediately fetched with a call to SignatureStatus.

23 KiB Raw Blame History

AppRequest

Request

Response

DocCreate

Request

Response

DocUpdate

Request

WriteQuery

FilePut

first Request : RandomAccessFilePut

first Response

one or more chunk put Request(s) : RandomAccessFilePutChunk

last Request : AddFile

final Response

FileGet

Request

first Response : FileMeta

more Responses : FileBinary

final response : EndOfStream

Subscribe

Request

first Response : AppTabInfo

second Response : AppState

followed by AppPatch

ReadQuery

Request

Response

RdfDump

Request

Response

History

Request

Response

SignatureStatus

Request

Response

SignatureRequest

Request

Response

SignedSnapshotRequest

Request

Response

23 KiB

Raw Blame History