NextGraph
/
oxigraph
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Activity

Updates BSBM benchmark

Browse Source
pull/46/head
Tpt 4 years ago
parent d9c36fb453
commit 4c2eafbfe0
6 changed files with 458 additions and 381 deletions
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Show Stats Download Patch File Download Diff File
  1. 6
      bench/README.md
  2. 804
      bench/bsbm.explore.svg
  3. 4
      bench/bsbm_blazegraph.sh
  4. 10
      bench/bsbm_graphdb.sh
  5. 13
      bench/bsbm_oxigraph.sh
  6. 2
      bench/bsbm_virtuoso.sh

6
bench/README.md
Unescape View File

@ -10,7 +10,7 @@ It provides a dataset generator and multiple set of queries grouped by "use case
We compare here Oxigraph with some existing SPARQL implementations (Blazegraph, Virtuoso and GraphDB). We compare here Oxigraph with some existing SPARQL implementations (Blazegraph, Virtuoso and GraphDB).
The dataset used in the following charts is generated with 10k "products" (see [its spec](http://wifo5-03.informatik.uni-mannheim.de/bizer/berlinsparqlbenchmark/spec/Dataset/index.html)). It leads to the creation of 3.5M triples. The dataset used in the following charts is generated with 10k "products" (see [its spec](http://wifo5-03.informatik.uni-mannheim.de/bizer/berlinsparqlbenchmark/spec/Dataset/index.html)). It leads to the creation of 3.5M triples.
It has been executed on a Dell Precision 5520 with 16GB of RAM. For Oxigraph, available memory has been limited to 1GB. It has been executed on a PrevailPro P3000 with 32GB of RAM.
### Explore ### Explore
The [explore use case](http://wifo5-03.informatik.uni-mannheim.de/bizer/berlinsparqlbenchmark/spec/ExploreUseCase/index.html) is composed of 11 queries that do simple data retrieval. The [explore use case](http://wifo5-03.informatik.uni-mannheim.de/bizer/berlinsparqlbenchmark/spec/ExploreUseCase/index.html) is composed of 11 queries that do simple data retrieval.
@ -19,6 +19,7 @@ Query 6 existed in previous versions of the benchmark as is now removed.
![explore use case results](bsbm.explore.svg) ![explore use case results](bsbm.explore.svg)
<!--
### Business Intelligence ### Business Intelligence
The [business intelligence use case](http://wifo5-03.informatik.uni-mannheim.de/bizer/berlinsparqlbenchmark/spec/BusinessIntelligenceUseCase/index.html) is composed of 8 complex analytics queries. The [business intelligence use case](http://wifo5-03.informatik.uni-mannheim.de/bizer/berlinsparqlbenchmark/spec/BusinessIntelligenceUseCase/index.html) is composed of 8 complex analytics queries.
@ -28,13 +29,14 @@ Oxigraph is still too slow to evaluate most of the queries.
It will be added in the graph after enough optimizations are done. It will be added in the graph after enough optimizations are done.
![explore use case results](bsbm.businessIntelligence.svg) ![explore use case results](bsbm.businessIntelligence.svg)
-->
## How to reproduce the benchmark ## How to reproduce the benchmark
The code of the benchmark is in the `bsbm-tools` submodule. You should pull it with a `git submodule update` before running the benchmark. The code of the benchmark is in the `bsbm-tools` submodule. You should pull it with a `git submodule update` before running the benchmark.
To run the benchmark for Oxigraph run `bash bsbm_oxigraph.sh`. It will compile the current Oxigraph code and run the benchmark against it. To run the benchmark for Oxigraph run `bash bsbm_oxigraph.sh`. It will compile the current Oxigraph code and run the benchmark against it.
You could tweak the number of products in the dataset and the available memory using the environment variables at the beginning of `bsbm_oxigraph.sh`. You could tweak the number of products in the dataset using the environment variables at the beginning of `bsbm_oxigraph.sh`.
To generate the plots run `python3 bsbsm-plot.py`. To generate the plots run `python3 bsbsm-plot.py`.

804
bench/bsbm.explore.svg
View File

File diff suppressed because it is too large Load Diff
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 44 KiB

After

Width:  |  Height:  |  Size: 46 KiB

4
bench/bsbm_blazegraph.sh
Unescape View File

@ -1,10 +1,10 @@
#!/usr/bin/env bash #!/usr/bin/env bash
DATASET_SIZE=100000 DATASET_SIZE=10000
cd bsbm-tools cd bsbm-tools
./generate -fc -pc ${DATASET_SIZE} -s nt -fn "explore-${DATASET_SIZE}" ./generate -fc -pc ${DATASET_SIZE} -s nt -fn "explore-${DATASET_SIZE}"
wget https://github.com/blazegraph/database/releases/download/BLAZEGRAPH_RELEASE_2_1_5/blazegraph.jar wget https://github.com/blazegraph/database/releases/download/BLAZEGRAPH_RELEASE_2_1_5/blazegraph.jar
/usr/lib/jvm/java-8-openjdk-amd64/bin/java -server -jar blazegraph.jar & /usr/lib/jvm/java-8-openjdk/bin/java -server -jar blazegraph.jar &
sleep 10 sleep 10
curl -f -X POST -H 'Content-Type:text/plain' --data-binary "@explore-${DATASET_SIZE}.nt" http://localhost:9999/blazegraph/sparql curl -f -X POST -H 'Content-Type:text/plain' --data-binary "@explore-${DATASET_SIZE}.nt" http://localhost:9999/blazegraph/sparql
./testdriver -ucf usecases/explore/sparql.txt -o "../bsbm.explore.blazegraph.${DATASET_SIZE}.2.1.5.xml" http://localhost:9999/blazegraph/sparql ./testdriver -ucf usecases/explore/sparql.txt -o "../bsbm.explore.blazegraph.${DATASET_SIZE}.2.1.5.xml" http://localhost:9999/blazegraph/sparql

10
bench/bsbm_graphdb.sh
Unescape View File

@ -1,19 +1,19 @@
#!/usr/bin/env bash #!/usr/bin/env bash
DATASET_SIZE=100000 DATASET_SIZE=10000
cd bsbm-tools cd bsbm-tools
./generate -fc -pc ${DATASET_SIZE} -s nt -fn "explore-${DATASET_SIZE}" ./generate -fc -pc ${DATASET_SIZE} -s nt -fn "explore-${DATASET_SIZE}"
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
../graphdb-free-8.11.0/bin/graphdb -s -Dgraphdb.logger.root.level=WARN & ../graphdb-free-9.3.3/bin/graphdb -s -Dgraphdb.logger.root.level=WARN &
sleep 10 sleep 10
curl -f -X POST http://localhost:7200/rest/repositories -H 'Content-Type:application/json' -d ' curl -f -X POST http://localhost:7200/rest/repositories -H 'Content-Type:application/json' -d '
{"id":"test","params":{"ruleset":{"label":"Ruleset","name":"ruleset","value":"empty"},"title":{"label":"Repository title","name":"title","value":"GraphDB Free repository"},"checkForInconsistencies":{"label":"Check for inconsistencies","name":"checkForInconsistencies","value":"false"},"disableSameAs":{"label":"Disable owl:sameAs","name":"disableSameAs","value":"true"},"baseURL":{"label":"Base URL","name":"baseURL","value":"http://example.org/owlim#"},"repositoryType":{"label":"Repository type","name":"repositoryType","value":"file-repository"},"id":{"label":"Repository ID","name":"id","value":"repo-test"},"storageFolder":{"label":"Storage folder","name":"storageFolder","value":"storage"}},"title":"Test","type":"free"} {"id":"test","params":{"ruleset":{"label":"Ruleset","name":"ruleset","value":"empty"},"title":{"label":"Repository title","name":"title","value":"GraphDB Free repository"},"checkForInconsistencies":{"label":"Check for inconsistencies","name":"checkForInconsistencies","value":"false"},"disableSameAs":{"label":"Disable owl:sameAs","name":"disableSameAs","value":"true"},"baseURL":{"label":"Base URL","name":"baseURL","value":"http://example.org/owlim#"},"repositoryType":{"label":"Repository type","name":"repositoryType","value":"file-repository"},"id":{"label":"Repository ID","name":"id","value":"repo-test"},"storageFolder":{"label":"Storage folder","name":"storageFolder","value":"storage"}},"title":"Test","type":"free"}
' '
curl -f -X PUT -H 'Content-Type:application/n-triples' --data-binary "@explore-${DATASET_SIZE}.nt" http://localhost:7200/repositories/test/statements curl -f -X PUT -H 'Content-Type:application/n-triples' --data-binary "@explore-${DATASET_SIZE}.nt" http://localhost:7200/repositories/test/statements
./testdriver -ucf usecases/explore/sparql.txt -o "../bsbm.explore.graphdb.${DATASET_SIZE}.8.11.0.xml" http://localhost:7200/repositories/test ./testdriver -ucf usecases/explore/sparql.txt -o "../bsbm.explore.graphdb.${DATASET_SIZE}.9.3.3.xml" http://localhost:7200/repositories/test
./testdriver -ucf usecases/businessIntelligence/sparql.txt -o "../bsbm.businessIntelligence.graphdb.${DATASET_SIZE}.8.11.0.xml" http://localhost:7200/repositories/test ./testdriver -ucf usecases/businessIntelligence/sparql.txt -o "../bsbm.businessIntelligence.graphdb.${DATASET_SIZE}.9.3.3.xml" http://localhost:7200/repositories/test
kill $! kill $!
sleep 5 sleep 5
rm -r ../graphdb-free-8.11.0/data rm -r ../graphdb-free-9.3.3/data
rm "explore-${DATASET_SIZE}.nt" rm "explore-${DATASET_SIZE}.nt"
rm -r td_data rm -r td_data

13
bench/bsbm_oxigraph.sh
Unescape View File

@ -1,19 +1,14 @@
#!/usr/bin/env bash #!/usr/bin/env bash
DATASET_SIZE=100000 # number of products in the dataset. There is around 350 triples generated by product. DATASET_SIZE=10000 # number of products in the dataset. There is around 350 triples generated by product.
MEMORY_SIZE=1000000 # available memory for Oxigraph in KB. Useful to simulate low RAM machines.
cd bsbm-tools cd bsbm-tools
./generate -fc -pc ${DATASET_SIZE} -s nt -fn "explore-${DATASET_SIZE}" ./generate -fc -pc ${DATASET_SIZE} -s nt -fn "explore-${DATASET_SIZE}"
cargo build --release --manifest-path="../../server/Cargo.toml" cargo build --release --manifest-path="../../server/Cargo.toml"
( ./../../target/release/oxigraph_server --file oxigraph_data --bind 127.0.0.1:7878 &
ulimit -d ${MEMORY_SIZE}
./../../target/release/oxigraph_server --file oxigraph_data --bind 127.0.0.1:7878
) &
sleep 5 sleep 5
curl -f -X POST -H 'Content-Type:application/n-triples' --data-binary "@explore-${DATASET_SIZE}.nt" http://127.0.0.1:7878/ curl -f -X POST -H 'Content-Type:application/n-triples' --data-binary "@explore-${DATASET_SIZE}.nt" http://127.0.0.1:7878/
./testdriver -ucf usecases/explore/sparql.txt -o "../bsbm.explore.oxigraph.${DATASET_SIZE}.${MEMORY_SIZE}.$(date +'%Y-%m-%d').xml" http://127.0.0.1:7878/query ./testdriver -ucf usecases/explore/sparql.txt -o "../bsbm.explore.oxigraph.${DATASET_SIZE}.0.1.0-rocksdb.xml" http://127.0.0.1:7878/query
./testdriver -ucf usecases/businessIntelligence/sparql.txt -o "../bsbm.businessIntelligence.oxigraph.${DATASET_SIZE}.${MEMORY_SIZE}.$(date +'%Y-%m-%d').xml" "http://127.0.0.1:7878/query" ./testdriver -ucf usecases/businessIntelligence/sparql.txt -o "../bsbm.businessIntelligence.${DATASET_SIZE}.0.1.0-rocksdb.xml" "http://127.0.0.1:7878/query"
kill $! kill $!
rm -r oxigraph_data rm -r oxigraph_data
rm "explore-${DATASET_SIZE}.nt" rm "explore-${DATASET_SIZE}.nt"

2
bench/bsbm_virtuoso.sh
Unescape View File

@ -1,6 +1,6 @@
#!/usr/bin/env bash #!/usr/bin/env bash
DATASET_SIZE=100000 DATASET_SIZE=10000
cd bsbm-tools cd bsbm-tools
./generate -fc -pc ${DATASET_SIZE} -s nt -fn "explore-${DATASET_SIZE}" ./generate -fc -pc ${DATASET_SIZE} -s nt -fn "explore-${DATASET_SIZE}"
cp ../virtuoso-opensource/database/virtuoso.ini.sample virtuoso.ini cp ../virtuoso-opensource/database/virtuoso.ini.sample virtuoso.ini

Write Preview
Loading…
Cancel
Save