From 9884decf527066fa86fad12b9cb9a56f3deaad37 Mon Sep 17 00:00:00 2001 From: Tpt Date: Thu, 7 Jan 2021 20:42:25 +0100 Subject: [PATCH] Splits the main README into a README per component --- README.md | 122 ++----------------------------------- js/README.md | 20 +++++- lib/Cargo.toml | 2 +- lib/README.md | 77 +++++++++++++++++++++++ lib/src/lib.rs | 7 ++- python/README.md | 19 +++++- python/docs/index.rst | 2 +- python/src/memory_store.rs | 8 ++- python/src/sled_store.rs | 10 ++- server/Cargo.toml | 2 +- server/README.md | 104 +++++++++++++++++++++++++++++++ wikibase/Cargo.toml | 2 +- wikibase/README.md | 81 ++++++++++++++++++++++++ 13 files changed, 324 insertions(+), 132 deletions(-) create mode 100644 lib/README.md create mode 100644 server/README.md create mode 100644 wikibase/README.md diff --git a/README.md b/README.md index 809eeee7..d411cd93 100644 --- a/README.md +++ b/README.md @@ -17,16 +17,16 @@ It also provides a set of utility functions for reading, writing, and processing Oxigraph is in heavy development and SPARQL query evaluation has not been optimized yet. It is split into multiple parts: -* The `lib` directory contains the database written as a Rust library. +* [The database written as a Rust library](https://crates.io/crates/oxigraph). Its source code is in the `lib` directory. [![Latest Version](https://img.shields.io/crates/v/oxigraph.svg)](https://crates.io/crates/oxigraph) [![Released API docs](https://docs.rs/oxigraph/badge.svg)](https://docs.rs/oxigraph) -* The `python` directory contains Pyoxigraph. Pyoxigraph allows using Oxigraph in Python. See the [Pyoxigraph website](https://oxigraph.org/pyoxigraph/) for Pyoxigraph documentation. [![PyPI](https://img.shields.io/pypi/v/pyoxigraph)](https://pypi.org/project/pyoxigraph/) -* The `js` directory contains bindings to use Oxigraph in JavaScript with the help of WebAssembly. See [its README](https://github.com/oxigraph/oxigraph/blob/master/js/README.md) for the JS bindings documentation. +* [`pyoxigraph` that exposes Oxigraph to the Python world](https://oxigraph.org/pyoxigraph/). Its source code is in the `python` directory. [![PyPI](https://img.shields.io/pypi/v/pyoxigraph)](https://pypi.org/project/pyoxigraph/) +* [JavaScript bindings for Oxigraph](https://www.npmjs.com/package/oxigraph). WebAssembly is used to package Oxigraph into a NodeJS compatible NPM package. Its source code is in the `js` directory. [![npm](https://img.shields.io/npm/v/oxigraph)](https://www.npmjs.com/package/oxigraph) -* The `server` directory contains a stand-alone binary of a web server implementing the [SPARQL 1.1 Protocol](https://www.w3.org/TR/sparql11-protocol/) and the [SPARQL 1.1 Graph Store Protocol](https://www.w3.org/TR/sparql11-http-rdf-update/). It uses the [RocksDB](https://rocksdb.org/) key-value store. +* [Oxigraph server](https://crates.io/crates/oxigraph_server) that provides a standalone binary of a web server implementing the [SPARQL 1.1 Protocol](https://www.w3.org/TR/sparql11-protocol/) and the [SPARQL 1.1 Graph Store Protocol](https://www.w3.org/TR/sparql11-http-rdf-update/). It uses the [RocksDB](https://rocksdb.org/) key-value store. Its source code is in the `server` directory. [![Latest Version](https://img.shields.io/crates/v/oxigraph_server.svg)](https://crates.io/crates/oxigraph_server) [![Docker Image Version (latest semver)](https://img.shields.io/docker/v/oxigraph/oxigraph?sort=semver)](https://hub.docker.com/repository/docker/oxigraph/oxigraph) -* The `wikibase` directory contains a stand-alone binary of a web server able to synchronize with a [Wikibase instance](https://wikiba.se/). +* [Oxigraph Wikibase](https://crates.io/crates/oxigraph_wikibase), a web server able to synchronize with a [Wikibase instance](https://wikiba.se/). Its source code is in the `wikibase` directory. [![Latest Version](https://img.shields.io/crates/v/oxigraph_wikibase.svg)](https://crates.io/crates/oxigraph_wikibase) [![Docker Image Version (latest semver)](https://img.shields.io/docker/v/oxigraph/oxigraph-wikibase?sort=semver)](https://hub.docker.com/repository/docker/oxigraph/oxigraph-wikibase) @@ -37,118 +37,6 @@ Oxigraph implements the following specifications: A preliminary benchmark [is provided](bench/README.md). -## Run the Web server - -### Installation - -You need to have [a recent stable version of Rust and Cargo installed](https://www.rust-lang.org/tools/install). You also need [clang](https://clang.llvm.org/) to build RocksDB. - -To download, build and install the latest released version run `cargo install oxigraph_server`. -There is no need to clone this git repository. - -To compile the server from source, clone this git repository, and execute `cargo build --release` in the `server` directory to compile the full server after having downloaded its dependencies. -It will create a fat binary in `target/release/oxigraph_server`. - -### Usage - -Run `oxigraph_server -f my_data_storage_directory` to start the server where `my_data_storage_directory` is the directory where you want Oxigraph data to be stored in. It listens by default on `localhost:7878`. - -The server provides an HTML UI with a form to execute SPARQL requests. - -It provides the following REST actions: -* `/query` allows to evaluate SPARQL queries against the server repository following the [SPARQL 1.1 Protocol](https://www.w3.org/TR/sparql11-protocol/#query-operation). - For example `curl -X POST -H 'Content-Type:application/sparql-query' --data 'SELECT * WHERE { ?s ?p ?o } LIMIT 10' http://localhost:7878/query`. - This action supports content negotiation and could return [Turtle](https://www.w3.org/TR/turtle/), [N-Triples](https://www.w3.org/TR/n-triples/), [RDF XML](https://www.w3.org/TR/rdf-syntax-grammar/), [SPARQL Query Results XML Format](http://www.w3.org/TR/rdf-sparql-XMLres/) and [SPARQL Query Results JSON Format](https://www.w3.org/TR/sparql11-results-json/). -* `/update` allows to execute SPARQL updates against the server repository following the [SPARQL 1.1 Protocol](https://www.w3.org/TR/sparql11-protocol/#update-operation). - For example `curl -X POST -H 'Content-Type: application/sparql-update' --data 'DELETE WHERE { ?p ?o }' http://localhost:7878/update`. -* `/store` allows to retrieve and change the server content using the [SPARQL 1.1 Graph Store HTTP Protocol](https://www.w3.org/TR/sparql11-http-rdf-update/). - For example `curl -f -X POST -H 'Content-Type:application/n-triples' --data-binary "@MY_FILE.nt" http://localhost:7878/store?graph=http://example.com/g` will add the N-Triples file MY_FILE.nt to the server dataset inside of the `http://example.com/g` named graph. - [Turtle](https://www.w3.org/TR/turtle/), [N-Triples](https://www.w3.org/TR/n-triples/) and [RDF XML](https://www.w3.org/TR/rdf-syntax-grammar/) are supported. - It is also possible to `POST`, `PUT` and `GET` the complete RDF dataset on the server using RDF dataset formats ([TriG](https://www.w3.org/TR/trig/) and [N-Quads](https://www.w3.org/TR/n-quads/)) against the `/store` endpoint. - For example `curl -f -X POST -H 'Content-Type:application/n-quads' --data-binary "@MY_FILE.nq" http://localhost:7878/store` will add the N-Quads file MY_FILE.nt to the server dataset. - -Use `oxigraph_server --help` to see the possible options when starting the server. - -### Using a Docker image - -#### Display the help menu -```sh -docker run --rm oxigraph/oxigraph --help -``` - -#### Run the Web server -Expose the server on port `7878` of the host machine, and save data on the local `./data` folder -```sh -docker run --init --rm -v $PWD/data:/data -p 7878:7878 oxigraph/oxigraph -b 0.0.0.0:7878 -f /data -``` - -You can then access it from your machine on port `7878`: -```sh -# Open the GUI in a browser -firefox http://localhost:7878 - -# Post some data -curl http://localhost:7878/store?default -H 'Content-Type: text/turtle' -d@./data.ttl - -# Make a query -curl -X POST -H 'Accept: application/sparql-results+json' -H 'Content-Type: application/sparql-query' --data 'SELECT * WHERE { ?s ?p ?o } LIMIT 10' http://localhost:7878/query - -# Make an UPDATE -curl -X POST -H 'Content-Type: application/sparql-update' --data 'DELETE WHERE { ?p ?o }' http://localhost:7878/update -``` - -You could easily build your own Docker image by running `docker build -t oxigraph server -f server/Dockerfile .` from the root directory. - -## Run the Wikibase server - -### Installation - -You need to have [a recent stable version of Rust and Cargo installed](https://www.rust-lang.org/tools/install). You also need [clang](https://clang.llvm.org/) to build RocksDB. - -To download, build and install the latest released version run `cargo install oxigraph_wikibase`. -There is no need to clone this git repository. - -To compile the server from source, clone this git repository, and execute `cargo build --release` in the `wikibase` directory to compile the full server after having downloaded its dependencies. -It will create a fat binary in `target/release/oxigraph_wikibase`. - -### Usage - -To start a server that is synchronized with [test.wikidata.org](https://test.wikidata.org) you should run: -```bash -./oxigraph_wikibase --mediawiki-api https://test.wikidata.org/w/api.php --mediawiki-base-url https://test.wikidata.org/wiki/ --namespaces 0,120 --file test.wikidata -``` - -It creates a SPARQL endpoint listening to `localhost:7878/query` that could be queried just like Blazegraph. - -The configuration parameters are: -* `mediawiki_api` URL of the MediaWiki API to use -* `mediawiki_base_url` Base URL of MediaWiki pages like `https://test.wikidata.org/wiki/` for test.wikidata.org or `http://localhost/w/index.php?title=` for "vanilla" installations. -* `namespaces` The ids of the Wikibase namespaces to synchronize with, separated by `,`. -* `file` Path of where Oxigraph should store its data. - - -You can then access it from your machine on port `7878`. No GUI is provided. -```sh -# Make a query -curl -X POST -H 'Accept: application/sparql-results+json' -H 'Content-Type: application/sparql-query' --data 'SELECT * WHERE { ?s ?p ?o } LIMIT 10' http://localhost:7878/query -``` -### Using a Docker image - -#### Display the help menu -```sh -docker run --rm oxigraph/oxigraph-wikibase --help -``` - -#### Run the Web server -Expose the server on port `7878` of the host machine, and save data on the local `./data` folder -```sh -docker run --init --rm -v $PWD/wikibase_data:/wikibase_data -p 7878:7878 oxigraph/oxigraph-wikibase -b 0.0.0.0:7878 -f /wikibase_data --mediawiki-api http://some.wikibase.instance/w/api.php --mediawiki-base-url http://some.wikibase.instance/wiki/ -``` - -Warning: the Wikibase instance needs to be accessible from within the container. -The clean way to do that could be to have both your wikibase and oxigraph_wikibase in the same [`docker-compose.yml`](https://docs.docker.com/compose/). - -You could easily build your own Docker image by running `docker build -t oxigraph-wikibase -f wikibase/Dockerfile .` from the root directory. ## License diff --git a/js/README.md b/js/README.md index ccf9e9e7..0269d8a2 100644 --- a/js/README.md +++ b/js/README.md @@ -5,7 +5,7 @@ Oxigraph for JavaScript [![actions status](https://github.com/oxigraph/oxigraph/workflows/build/badge.svg)](https://github.com/oxigraph/oxigraph/actions) [![Gitter](https://badges.gitter.im/oxigraph/community.svg)](https://gitter.im/oxigraph/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge) -This package provides a JavaScript API on top of Oxigraph compiled with WebAssembly. +This package provides a JavaScript API on top of [Oxigraph](https://crates.io/crates/oxigraph), compiled with WebAssembly. Oxigraph is a graph database written in Rust implementing the [SPARQL](https://www.w3.org/TR/sparql11-overview/) standard. @@ -185,7 +185,6 @@ Example of building a Turtle file from the named graph `"] license = "MIT OR Apache-2.0" -readme = "../README.md" +readme = "README.md" keywords = ["RDF", "SPARQL", "graph-database", "database"] categories = ["database-implementations"] repository = "https://github.com/oxigraph/oxigraph" diff --git a/lib/README.md b/lib/README.md new file mode 100644 index 00000000..afb27c9a --- /dev/null +++ b/lib/README.md @@ -0,0 +1,77 @@ +Oxigraph +======== + +[![Latest Version](https://img.shields.io/crates/v/oxigraph.svg)](https://crates.io/crates/oxigraph) +[![Released API docs](https://docs.rs/oxigraph/badge.svg)](https://docs.rs/oxigraph) +[![Crates.io downloads](https://img.shields.io/crates/d/oxigraph)](https://crates.io/crates/oxigraph) +[![actions status](https://github.com/oxigraph/oxigraph/workflows/build/badge.svg)](https://github.com/oxigraph/oxigraph/actions) +[![Gitter](https://badges.gitter.im/oxigraph/community.svg)](https://gitter.im/oxigraph/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge) + +Oxigraph is a graph database library implementing the [SPARQL](https://www.w3.org/TR/sparql11-overview/) standard. + +Its goal is to provide a compliant, safe and fast graph database. +It also provides a set of utility functions for reading, writing, and processing RDF files. + +It currently provides three store implementations providing [SPARQL](https://www.w3.org/TR/sparql11-overview/) capability: +* `MemoryStore`: a simple in memory implementation. +* `RocksDbStore`: a file system implementation based on the [RocksDB](https://rocksdb.org/) key-value store. + It requires the `"rocksdb"` feature to be activated. + The [clang](https://clang.llvm.org/) compiler needs to be installed to compile RocksDB. +* `SledStore`: another file system implementation based on the [Sled](https://sled.rs/) key-value store. + It requires the `"sled"` feature to be activated. + Sled is much faster to build than RockDB and does not require a C++ compiler. + However, Sled is still in development, less tested and data load seems much slower than RocksDB. + +Oxigraph is in heavy development and SPARQL query evaluation has not been optimized yet. + +The disabled by default `"sophia"` feature provides [`sophia_api`](https://docs.rs/sophia_api/) traits implementation on Oxigraph terms and stores. + +Oxigraph also provides [a standalone HTTP server](https://crates.io/crates/oxigraph_server) and [a Python library](https://oxigraph.org/pyoxigraph/) based on this library. + + +Oxigraph implements the following specifications: +* [SPARQL 1.1 Query](https://www.w3.org/TR/sparql11-query/), [SPARQL 1.1 Update](https://www.w3.org/TR/sparql11-update/), and [SPARQL 1.1 Federated Query](https://www.w3.org/TR/sparql11-federated-query/). +* [Turtle](https://www.w3.org/TR/turtle/), [TriG](https://www.w3.org/TR/trig/), [N-Triples](https://www.w3.org/TR/n-triples/), [N-Quads](https://www.w3.org/TR/n-quads/), and [RDF XML](https://www.w3.org/TR/rdf-syntax-grammar/) RDF serialization formats for both data ingestion and retrieval using the [Rio library](https://github.com/oxigraph/rio). +* [SPARQL Query Results XML Format](http://www.w3.org/TR/rdf-sparql-XMLres/), [SPARQL 1.1 Query Results JSON Format](https://www.w3.org/TR/sparql11-results-json/) and [SPARQL 1.1 Query Results CSV and TSV Formats](https://www.w3.org/TR/sparql11-results-csv-tsv/). + +A preliminary benchmark [is provided](../bench/README.md). + +Usage example with the `MemoryStore`: + +```rust +use oxigraph::MemoryStore; +use oxigraph::model::*; +use oxigraph::sparql::QueryResults; + +let store = MemoryStore::new(); + +// insertion +let ex = NamedNode::new("http://example.com")?; +let quad = Quad::new(ex.clone(), ex.clone(), ex.clone(), None); +store.insert(quad.clone()); + +// quad filter +let results: Vec = store.quads_for_pattern(Some(ex.as_ref().into()), None, None, None).collect(); +assert_eq!(vec![quad], results); + +// SPARQL query +if let QueryResults::Solutions(mut solutions) = store.query("SELECT ?s WHERE { ?s ?p ?o }")? { + assert_eq!(solutions.next().unwrap()?.get("s"), Some(&ex.into())); +} +``` + +## License + +This project is licensed under either of + + * Apache License, Version 2.0, ([LICENSE-APACHE](../LICENSE-APACHE) or + http://www.apache.org/licenses/LICENSE-2.0) + * MIT license ([LICENSE-MIT](../LICENSE-MIT) or + http://opensource.org/licenses/MIT) + +at your option. + + +### Contribution + +Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in Futures by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions. diff --git a/lib/src/lib.rs b/lib/src/lib.rs index 9cc04e01..b6180d02 100644 --- a/lib/src/lib.rs +++ b/lib/src/lib.rs @@ -1,6 +1,7 @@ -//! Oxigraph is a graph database implementing the [SPARQL](https://www.w3.org/TR/sparql11-overview/) standard. +//! Oxigraph is a graph database library implementing the [SPARQL](https://www.w3.org/TR/sparql11-overview/) standard. //! //! Its goal is to provide a compliant, safe and fast graph database. +//! It also provides a set of utility functions for reading, writing, and processing RDF files. //! //! It currently provides three store implementations providing [SPARQL](https://www.w3.org/TR/sparql11-overview/) capability: //! * [`MemoryStore`](store::memory::MemoryStore): a simple in memory implementation. @@ -12,10 +13,12 @@ //! Sled is much faster to build than RockDB and does not require a C++ compiler. //! However, Sled is still in developpment, less tested and data load seems much slower than RocksDB. //! -//! Oxigraph also provides a set of utility functions for reading, writing and processing RDF files. +//! Oxigraph is in heavy development and SPARQL query evaluation has not been optimized yet. //! //! The disabled by default `"sophia"` feature provides [`sophia_api`](https://docs.rs/sophia_api/) traits implemention on Oxigraph terms and stores. //! +//! Oxigraph also provides [a standalone HTTP server](https://crates.io/crates/oxigraph_server) based on this library. +//! //! Usage example with the [`MemoryStore`](store::memory::MemoryStore): //! //! ``` diff --git a/python/README.md b/python/README.md index 005f264e..ca14230c 100644 --- a/python/README.md +++ b/python/README.md @@ -8,7 +8,7 @@ Pyoxigraph (Oxigraph for Python) [![Gitter](https://badges.gitter.im/oxigraph/community.svg)](https://gitter.im/oxigraph/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge) Pyoxigraph is a graph database library implementing the [SPARQL](https://www.w3.org/TR/sparql11-overview/) standard. -It is a Python library written on top of [Oxigraph](https://github.com/oxigraph/oxigraph). +It is a Python library written on top of [Oxigraph](https://crates.io/crates/oxigraph). Pyoxigraph offers two stores with [SPARQL 1.1](https://www.w3.org/TR/sparql11-overview/) capabilities. One of the store is in-memory, and the other one is disk based. @@ -32,7 +32,6 @@ Pyoxigraph documentation is [available on the Oxigraph website](https://oxigraph To build and install the development version of pyoxigraph you need to clone this git repository and to run `pip install .` in the `python` directory (the one this README is in). - ## How to contribute Pyoxigraph is written in Rust using [PyO3](https://github.com/PyO3/pyo3). @@ -43,3 +42,19 @@ To install a development version of Oxigraph just run `maturin develop` in this The Python bindings tests are written in Python. To run them use `python -m unittest` in the `tests` directory. + +## License + +This project is licensed under either of + +* Apache License, Version 2.0, ([LICENSE-APACHE](../LICENSE-APACHE) or + http://www.apache.org/licenses/LICENSE-2.0) +* MIT license ([LICENSE-MIT](../LICENSE-MIT) or + http://opensource.org/licenses/MIT) + +at your option. + + +### Contribution + +Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in Futures by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions. diff --git a/python/docs/index.rst b/python/docs/index.rst index 2ab66625..7d5f66b9 100644 --- a/python/docs/index.rst +++ b/python/docs/index.rst @@ -14,7 +14,7 @@ pyoxigraph |release| Pyoxigraph is a Python graph database library implementing the `SPARQL `_ standard. -It is built on top of `Oxigraph `_ using `PyO3 `_. +It is built on top of `Oxigraph `_ using `PyO3 `_. It offers two stores with `SPARQL 1.1 `_ capabilities. One of the store is in-memory, and the other one is disk based. diff --git a/python/src/memory_store.rs b/python/src/memory_store.rs index bcce09a3..5202f38f 100644 --- a/python/src/memory_store.rs +++ b/python/src/memory_store.rs @@ -17,7 +17,6 @@ use std::io::BufReader; /// In-memory store. /// It encodes a `RDF dataset `_ and allows to query it using SPARQL. /// -/// /// The :py:func:`str` function provides a serialization of the store in NQuads: /// /// >>> store = MemoryStore() @@ -337,6 +336,9 @@ impl PyMemoryStore { /// Returns an iterator over all the store named graphs /// + /// :return: an iterator of the store graph names + /// :rtype: iter(NamedNode or BlankNode) + /// /// >>> store = MemoryStore() /// >>> store.add(Quad(NamedNode('http://example.com'), NamedNode('http://example.com/p'), Literal('1'), NamedNode('http://example.com/g'))) /// >>> list(store.named_graphs()) @@ -350,7 +352,7 @@ impl PyMemoryStore { /// Adds a named graph to the store /// - /// :param graph_name: the quad to add + /// :param graph_name: the name of the name graph to add /// :type graph_name: NamedNode or BlankNode /// /// >>> store = MemoryStore() @@ -370,7 +372,7 @@ impl PyMemoryStore { /// /// The default graph will not be remove but just cleared. /// - /// :param graph_name: the quad to add + /// :param graph_name: the name of the name graph to remove /// :type graph_name: NamedNode or BlankNode or DefaultGraph /// /// >>> store = MemoryStore() diff --git a/python/src/sled_store.rs b/python/src/sled_store.rs index 82621d67..76ca1c89 100644 --- a/python/src/sled_store.rs +++ b/python/src/sled_store.rs @@ -355,6 +355,10 @@ impl PySledStore { /// Returns an iterator over all the store named graphs /// + /// :return: an iterator of the store graph names + /// :rtype: iter(NamedNode or BlankNode) + /// :raises IOError: if an I/O error happens during the named graphs lookup + /// /// >>> store = MemoryStore() /// >>> store.add(Quad(NamedNode('http://example.com'), NamedNode('http://example.com/p'), Literal('1'), NamedNode('http://example.com/g'))) /// >>> list(store.named_graphs()) @@ -368,8 +372,9 @@ impl PySledStore { /// Adds a named graph to the store /// - /// :param graph_name: the quad to add + /// :param graph_name: the name of the name graph to add /// :type graph_name: NamedNode or BlankNode + /// :raises IOError: if an I/O error happens during the named graph insertion /// /// >>> store = MemoryStore() /// >>> store.add_graph(NamedNode('http://example.com/g')) @@ -393,8 +398,9 @@ impl PySledStore { /// /// The default graph will not be remove but just cleared. /// - /// :param graph_name: the quad to add + /// :param graph_name: the name of the name graph to remove /// :type graph_name: NamedNode or BlankNode or DefaultGraph + /// :raises IOError: if an I/O error happens during the named graph removal /// /// >>> store = MemoryStore() /// >>> quad = Quad(NamedNode('http://example.com'), NamedNode('http://example.com/p'), Literal('1'), NamedNode('http://example.com/g')) diff --git a/server/Cargo.toml b/server/Cargo.toml index b37f9694..215b3a45 100644 --- a/server/Cargo.toml +++ b/server/Cargo.toml @@ -3,7 +3,7 @@ name = "oxigraph_server" version = "0.1.1" authors = ["Tpt "] license = "MIT OR Apache-2.0" -readme = "../README.md" +readme = "README.md" repository = "https://github.com/oxigraph/oxigraph" description = """ SPARQL server based on Oxigraph diff --git a/server/README.md b/server/README.md new file mode 100644 index 00000000..8ea1786c --- /dev/null +++ b/server/README.md @@ -0,0 +1,104 @@ +Oxigraph Server +=============== + +[![Latest Version](https://img.shields.io/crates/v/oxigraph_server.svg)](https://crates.io/crates/oxigraph_server) +[![Crates.io downloads](https://img.shields.io/crates/d/oxigraph_server)](https://crates.io/crates/oxigraph_server) +[![Docker Image Version (latest semver)](https://img.shields.io/docker/v/oxigraph/oxigraph?sort=semver)](https://hub.docker.com/repository/docker/oxigraph/oxigraph) +[![Docker Image Size (latest semver)](https://img.shields.io/docker/image-size/oxigraph/oxigraph)](https://hub.docker.com/repository/docker/oxigraph/oxigraph) +[![Docker Pulls](https://img.shields.io/docker/pulls/oxigraph/oxigraph)](https://hub.docker.com/repository/docker/oxigraph/oxigraph) +[![actions status](https://github.com/oxigraph/oxigraph/workflows/build/badge.svg)](https://github.com/oxigraph/oxigraph/actions) +[![Gitter](https://badges.gitter.im/oxigraph/community.svg)](https://gitter.im/oxigraph/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge) + +Oxigraph Server is a standalone HTTP server providing a graph database implementing the [SPARQL](https://www.w3.org/TR/sparql11-overview/) standard. + +Its goal is to provide a compliant, safe, and fast graph database based on the [RocksDB](https://rocksdb.org/) key-value stores. +It is written in Rust. +It also provides a set of utility functions for reading, writing, and processing RDF files. + +Oxigraph is in heavy development and SPARQL query evaluation has not been optimized yet. + +It is also usable as [a Rust library](https://crates.io/crates/oxigraph) and as [a Python library](https://oxigraph.org/pyoxigraph/). + +Oxigraph implements the following specifications: +* [SPARQL 1.1 Query](https://www.w3.org/TR/sparql11-query/), [SPARQL 1.1 Update](https://www.w3.org/TR/sparql11-update/), and [SPARQL 1.1 Federated Query](https://www.w3.org/TR/sparql11-federated-query/). +* [Turtle](https://www.w3.org/TR/turtle/), [TriG](https://www.w3.org/TR/trig/), [N-Triples](https://www.w3.org/TR/n-triples/), [N-Quads](https://www.w3.org/TR/n-quads/), and [RDF XML](https://www.w3.org/TR/rdf-syntax-grammar/) RDF serialization formats for both data ingestion and retrieval using the [Rio library](https://github.com/oxigraph/rio). +* [SPARQL Query Results XML Format](http://www.w3.org/TR/rdf-sparql-XMLres/), [SPARQL 1.1 Query Results JSON Format](https://www.w3.org/TR/sparql11-results-json/) and [SPARQL 1.1 Query Results CSV and TSV Formats](https://www.w3.org/TR/sparql11-results-csv-tsv/). +* [SPARQL 1.1 Protocol](https://www.w3.org/TR/sparql11-protocol/#query-operation) and [SPARQL 1.1 Graph Store HTTP Protocol](https://www.w3.org/TR/sparql11-http-rdf-update/). + +A preliminary benchmark [is provided](../bench/README.md). + +## Installation + +You need to have [a recent stable version of Rust and Cargo installed](https://www.rust-lang.org/tools/install). You also need [clang](https://clang.llvm.org/) to build RocksDB. + +To download, build and install the latest released version run `cargo install oxigraph_server`. +There is no need to clone the git repository. + +To compile the server from source, clone this git repository, and execute `cargo build --release` in the `server` directory to compile the full server after having downloaded its dependencies. +It will create a fat binary in `target/release/oxigraph_server`. + +## Usage + +Run `oxigraph_server -f my_data_storage_directory` to start the server where `my_data_storage_directory` is the directory where you want Oxigraph data to be stored in. It listens by default on `localhost:7878`. + +The server provides an HTML UI with a form to execute SPARQL requests. + +It provides the following REST actions: +* `/query` allows to evaluate SPARQL queries against the server repository following the [SPARQL 1.1 Protocol](https://www.w3.org/TR/sparql11-protocol/#query-operation). + For example `curl -X POST -H 'Content-Type:application/sparql-query' --data 'SELECT * WHERE { ?s ?p ?o } LIMIT 10' http://localhost:7878/query`. + This action supports content negotiation and could return [Turtle](https://www.w3.org/TR/turtle/), [N-Triples](https://www.w3.org/TR/n-triples/), [RDF XML](https://www.w3.org/TR/rdf-syntax-grammar/), [SPARQL Query Results XML Format](http://www.w3.org/TR/rdf-sparql-XMLres/) and [SPARQL Query Results JSON Format](https://www.w3.org/TR/sparql11-results-json/). +* `/update` allows to execute SPARQL updates against the server repository following the [SPARQL 1.1 Protocol](https://www.w3.org/TR/sparql11-protocol/#update-operation). + For example `curl -X POST -H 'Content-Type: application/sparql-update' --data 'DELETE WHERE { ?p ?o }' http://localhost:7878/update`. +* `/store` allows to retrieve and change the server content using the [SPARQL 1.1 Graph Store HTTP Protocol](https://www.w3.org/TR/sparql11-http-rdf-update/). + For example `curl -f -X POST -H 'Content-Type:application/n-triples' --data-binary "@MY_FILE.nt" http://localhost:7878/store?graph=http://example.com/g` will add the N-Triples file MY_FILE.nt to the server dataset inside of the `http://example.com/g` named graph. + [Turtle](https://www.w3.org/TR/turtle/), [N-Triples](https://www.w3.org/TR/n-triples/) and [RDF XML](https://www.w3.org/TR/rdf-syntax-grammar/) are supported. + It is also possible to `POST`, `PUT` and `GET` the complete RDF dataset on the server using RDF dataset formats ([TriG](https://www.w3.org/TR/trig/) and [N-Quads](https://www.w3.org/TR/n-quads/)) against the `/store` endpoint. + For example `curl -f -X POST -H 'Content-Type:application/n-quads' --data-binary "@MY_FILE.nq" http://localhost:7878/store` will add the N-Quads file MY_FILE.nq to the server dataset. + +Use `oxigraph_server --help` to see the possible options when starting the server. + +## Using a Docker image + +### Display the help menu +```sh +docker run --rm oxigraph/oxigraph --help +``` + +### Run the Web server +Expose the server on port `7878` of the host machine, and save data on the local `./data` folder +```sh +docker run --init --rm -v $PWD/data:/data -p 7878:7878 oxigraph/oxigraph -b 0.0.0.0:7878 -f /data +``` + +You can then access it from your machine on port `7878`: +```sh +# Open the GUI in a browser +firefox http://localhost:7878 + +# Post some data +curl http://localhost:7878/store?default -H 'Content-Type: text/turtle' -d@./data.ttl + +# Make a query +curl -X POST -H 'Accept: application/sparql-results+json' -H 'Content-Type: application/sparql-query' --data 'SELECT * WHERE { ?s ?p ?o } LIMIT 10' http://localhost:7878/query + +# Make an UPDATE +curl -X POST -H 'Content-Type: application/sparql-update' --data 'DELETE WHERE { ?p ?o }' http://localhost:7878/update +``` + +You could easily build your own Docker image by running `docker build -t oxigraph server -f server/Dockerfile .` from the root directory. + +## License + +This project is licensed under either of + +* Apache License, Version 2.0, ([LICENSE-APACHE](../LICENSE-APACHE) or + http://www.apache.org/licenses/LICENSE-2.0) +* MIT license ([LICENSE-MIT](../LICENSE-MIT) or + http://opensource.org/licenses/MIT) + +at your option. + + +### Contribution + +Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in Futures by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions. diff --git a/wikibase/Cargo.toml b/wikibase/Cargo.toml index 856aee27..33a47e8e 100644 --- a/wikibase/Cargo.toml +++ b/wikibase/Cargo.toml @@ -3,7 +3,7 @@ name = "oxigraph_wikibase" version = "0.1.1" authors = ["Tpt "] license = "MIT OR Apache-2.0" -readme = "../README.md" +readme = "README.md" repository = "https://github.com/oxigraph/oxigraph" description = """ SPARQL server based on Oxigraph for Wikibase instances diff --git a/wikibase/README.md b/wikibase/README.md new file mode 100644 index 00000000..c3b16236 --- /dev/null +++ b/wikibase/README.md @@ -0,0 +1,81 @@ +Oxigraph Wikibase +================= + +[![Latest Version](https://img.shields.io/crates/v/oxigraph_wikibase.svg)](https://crates.io/crates/oxigraph_wikibase) +[![Crates.io](https://img.shields.io/crates/d/oxigraph_wikibase)](https://crates.io/crates/oxigraph_wikibase) +[![Docker Image Version (latest semver)](https://img.shields.io/docker/v/oxigraph/oxigraph-wikibase?sort=semver)](https://hub.docker.com/repository/docker/oxigraph/oxigraph-wikibase) +[![Docker Image Size (latest semver)](https://img.shields.io/docker/image-size/oxigraph/oxigraph-wikibase)](https://hub.docker.com/repository/docker/oxigraph/oxigraph-wikibase) +[![Docker Pulls](https://img.shields.io/docker/pulls/oxigraph/oxigraph-wikibase)](https://hub.docker.com/repository/docker/oxigraph/oxigraph-wikibase) +[![actions status](https://github.com/oxigraph/oxigraph/workflows/build/badge.svg)](https://github.com/oxigraph/oxigraph/actions) +[![Gitter](https://badges.gitter.im/oxigraph/community.svg)](https://gitter.im/oxigraph/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge) + +Oxigraph Wikibase is a [SPARQL](https://www.w3.org/TR/sparql11-overview/) web server able to synchronize with a [Wikibase instance](https://wikiba.se/). +It is based on [Oxigraph](https://crates.io/crates/oxigraph). + +Oxigraph and Oxigraph Wikibase are in heavy development and not been optimized yet. + +## Installation + +You need to have [a recent stable version of Rust and Cargo installed](https://www.rust-lang.org/tools/install). You also need [clang](https://clang.llvm.org/) to build RocksDB. + +To download, build and install the latest released version run `cargo install oxigraph_wikibase`. +There is no need to clone the git repository. + +To compile the server from source, clone this git repository, and execute `cargo build --release` in the `wikibase` directory to compile the full server after having downloaded its dependencies. +It will create a fat binary in `target/release/oxigraph_wikibase`. + +## Usage + +To start a server that is synchronized with [test.wikidata.org](https://test.wikidata.org) you should run: +```bash +./oxigraph_wikibase --mediawiki-api https://test.wikidata.org/w/api.php --mediawiki-base-url https://test.wikidata.org/wiki/ --namespaces 0,120 --file test.wikidata +``` + +It creates a SPARQL endpoint listening to `localhost:7878/query` that could be queried just like Blazegraph. + +The configuration parameters are: +* `mediawiki_api` URL of the MediaWiki API to use +* `mediawiki_base_url` Base URL of MediaWiki pages like `https://test.wikidata.org/wiki/` for test.wikidata.org or `http://localhost/w/index.php?title=` for "vanilla" installations. +* `namespaces` The ids of the Wikibase namespaces to synchronize with, separated by `,`. +* `file` Path of where Oxigraph should store its data. + + +You can then access it from your machine on port `7878`. No GUI is provided. +```sh +# Make a query +curl -X POST -H 'Accept: application/sparql-results+json' -H 'Content-Type: application/sparql-query' --data 'SELECT * WHERE { ?s ?p ?o } LIMIT 10' http://localhost:7878/query +``` +## Using a Docker image + +### Display the help menu +```sh +docker run --rm oxigraph/oxigraph-wikibase --help +``` + +### Run the Web server +Expose the server on port `7878` of the host machine, and save data on the local `./data` folder +```sh +docker run --init --rm -v $PWD/wikibase_data:/wikibase_data -p 7878:7878 oxigraph/oxigraph-wikibase -b 0.0.0.0:7878 -f /wikibase_data --mediawiki-api http://some.wikibase.instance/w/api.php --mediawiki-base-url http://some.wikibase.instance/wiki/ +``` + +Warning: the Wikibase instance needs to be accessible from within the container. +The clean way to do that could be to have both your wikibase and oxigraph_wikibase in the same [`docker-compose.yml`](https://docs.docker.com/compose/). + +You could easily build your own Docker image by running `docker build -t oxigraph-wikibase -f wikibase/Dockerfile .` from the root directory. + + +## License + +This project is licensed under either of + +* Apache License, Version 2.0, ([LICENSE-APACHE](../LICENSE-APACHE) or + http://www.apache.org/licenses/LICENSE-2.0) +* MIT license ([LICENSE-MIT](../LICENSE-MIT) or + http://opensource.org/licenses/MIT) + +at your option. + + +### Contribution + +Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in Futures by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.