Reports¶
PHPBench includes a reporting framework. Report
generators provide report data which can subsequently be
rendered by report renderer (e.g. console
or html
).
This guide will deal with generating reports and assume that the console
renderer is used.
Generating Reports¶
To report after a benchmarking run:
phpbench run --report=aggregate
Multiple reports can be specified:
phpbench run --report=aggregate --report=env
You can also tag runs and generate reports afterwards:
phpbench run --tag=main
phpbench report --ref=main --report=aggregate
For more information on storage see storage.
Configuring Reports¶
All reports can be configured either in the report configuration or directly on the command line using a simplified JSON encoded string instead of the report name:
To configure a report in phpbench.json
:
{
"runner.path": "NothingBench.php",
"runner.executor": "debug",
"report.generators": {
"my-report": {
"generator": "expression",
"cols": [ "benchmark", "tag", "subject", "mean" ],
"break": ["benchmark"]
}
}
}
Then run it with:
phpbench run --report=my-report
You can also configure reports directly from the command line using simplified JSON:
phpbench run --report='"generator": "expression", "cols": [ "benchmark", "tag", "subject", "mean" ], "break": ["benchmark"]'
In each case it is required to specify the generator
key which corresponds
to the registered name of the report generator.
You may also extend an existing report configuration:
phpbench run --report='extends: aggregate, break: ["benchmark", "set"]'
This will merge the given keys onto the configuration for the aggregate report.
Default Reports¶
aggregate
¶
Shows aggregate details of each set of iterations.
Run:
phpbench run --report=aggregate
And get something like:
+--------------+--------------+-----+------+-----+----------+----------+--------+
| benchmark | subject | set | revs | its | mem_peak | mode | rstdev |
+--------------+--------------+-----+------+-----+----------+----------+--------+
| NothingBench | benchNothing | | 1 | 1 | 100b | 10.000μs | ±0.00% |
+--------------+--------------+-----+------+-----+----------+----------+--------+
It is uses the table
generator, see Expression Report for more information.
benchmark
¶
Sumarize each benchmark:
Run:
phpbench run --report=benchmark --iterations=4
And get something like:
NothingBench
============
Average iteration times by variant
10.0μs │ █
8.8μs │ █
7.5μs │ █
6.3μs │ █
5.0μs │ █
3.8μs │ █
2.5μs │ █
1.3μs │ █
└───
1
[█ <current>]
1: benchNothing
Memory by variant
100b │ █
88b │ █
75b │ █
63b │ █
50b │ █
38b │ █
25b │ █
13b │ █
└───
1
[█ <current>]
1: benchNothing
<current>
+-----------------+--------+----------+--------+---------+
| subject | memory | mode | rstdev | stdev |
+-----------------+--------+----------+--------+---------+
| benchNothing () | 100b | 10.000μs | ±0.00% | 0.000μs |
+-----------------+--------+----------+--------+---------+
default
¶
The default report presents the result of each iteration:
Run:
phpbench run --report=default --iterations=4
And get something like:
+------+--------------+--------------+-----+------+----------+----------+--------------+----------------+
| iter | benchmark | subject | set | revs | mem_peak | time_avg | comp_z_value | comp_deviation |
+------+--------------+--------------+-----+------+----------+----------+--------------+----------------+
| 0 | NothingBench | benchNothing | | 1 | 100b | 10.000μs | +0.00σ | +0.00% |
| 1 | NothingBench | benchNothing | | 1 | 100b | 10.000μs | +0.00σ | +0.00% |
| 2 | NothingBench | benchNothing | | 1 | 100b | 10.000μs | +0.00σ | +0.00% |
| 3 | NothingBench | benchNothing | | 1 | 100b | 10.000μs | +0.00σ | +0.00% |
+------+--------------+--------------+-----+------+----------+----------+--------------+----------------+
It is uses the table
generator, see Expression Report for more information.
env
¶
This report shows information about the environment that the benchmarks were executed in.
Run:
phpbench run --report=env
And get something like:
Suite E3X6AeMdP7L9E7E0X0A7McP1L8E1EdXbAbMbP7La xxxx-xx-xx xx-xx-xx
+----------+----------+-------+
| provider | key | value |
+----------+----------+-------+
| test | example1 | 1 |
| test | example2 | 2 |
+----------+----------+-------+
Generator: Env.
Columns:
provider: Name of the environment provider (see
PhpBench\Environment\Provider
in the code for more information).key: Information key.
value: Information value.
See the Environment reference for more information.
Note
The information available will differ depending on platform. For example,
unix-sysload
is only available on UNIX platforms, where
as the VCS field will appear only when a supported VCS system is being
used.
overview
¶
Shows overview of a single or mutiple runs:
Run:
phpbench run --report=overview --iterations=4
And get something like:
Overview
========
Suites
+-----------+---------------------+------+------------+--------+------------+------+----------+----------+
| suite | date | php | vcs branch | xdebug | iterations | revs | mode | net_time |
+-----------+---------------------+------+------------+--------+------------+------+----------+----------+
| <current> | xxxx-xx-xx xx-xx-xx | null | null | null | 4 | 4 | 10.000μs | 40.000μs |
+-----------+---------------------+------+------------+--------+------------+------+----------+----------+
Average iteration times aggregated by benchmark
10.0μs │ █
8.8μs │ █
7.5μs │ █
6.3μs │ █
5.0μs │ █
3.8μs │ █
2.5μs │ █
1.3μs │ █
└───
1
[█ <current>]
1: NothingBench
Average peak memory aggregated by benchmark
100b │ █
88b │ █
75b │ █
63b │ █
50b │ █
38b │ █
25b │ █
13b │ █
└───
1
[█ <current>]
1: NothingBench
By Benchmark
10.0μs │ █
8.8μs │ █
7.5μs │ █
6.3μs │ █
5.0μs │ █
3.8μs │ █
2.5μs │ █
1.3μs │ █
└───
1
[█ NothingBench]
1: <current>
Table
=====
<current>
+--------------+-----------------+--------+----------+----------+----------+--------+---------+
| benchmark | subject | memory | min | max | mode | rstdev | stdev |
+--------------+-----------------+--------+----------+----------+----------+--------+---------+
| NothingBench | benchNothing () | 100b | 10.000μs | 10.000μs | 10.000μs | ±0.00% | 0.000μs |
+--------------+-----------------+--------+----------+----------+----------+--------+---------+