Summary:
Recently merged CI benchmark scripts were failing.
There has clearly been a major revision of the fields of benchmark output. The upload script expects and sanity-checks the existence of some fields (changes date to conform to OpenSearch format)..., so the script needs to change.
Also add a bit more exception checking to make it more obvious when this happens again.
We have deleted the existing report.tsv from the benchmark machine. An existing report.tsv is appended to by default, so that if the fields change, later rows no longer match the header. This makes for an upload that dies half way through the report file, when the format no longer matches the header.
Re-instate the config.yml for running the benchmarks, so we can once again test it in situ.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/10120
Reviewed By: pdillinger
Differential Revision: D37314908
Pulled By: jay-zhuang
fbshipit-source-id: 34f5243fee694b75c6838eb55d3398e4273254b2
Summary:
CircleCI runner based benchmarking. A runner is a dedicate machine configured for CircleCI to perform work on. Our work is a repeatable benchmark, the `benchmark-linux` job in `config.yml`
A runner, in CircleCI terminology, is a machine that is managed by the client (us) rather than running on CircleCI resources in the cloud. This means that we define and configure the iron, and that therefore the performance is repeatable and predictable. Which is what we need for performance regression benchmarking.
On a time schedule (or on commit, during branch development) benchmarks are set off on the runner, and then a script is run `benchmark_log_tool.py` which parses the benchmark output and pushes it into a pre-configured OpenSearch document connected to an OpenSearch dashboard. Members of the team can examine benchmark performance changes on the dashboard.
As time progresses we can add different benchmarks to the suite which gets run.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9723
Reviewed By: pdillinger
Differential Revision: D35555626
Pulled By: jay-zhuang
fbshipit-source-id: c6a905ca04494495c3784cfbb991f5ab90c807ee
Bo Wang