Reports

PHPBench includes a reporting framework. Report generators provide report data which can subsequently be rendered by report renderer (e.g. console or html).

This guide will deal with generating reports and assume that the console renderer is used.

Generating Reports

To report after a benchmarking run:

phpbench run --report=aggregate

Multiple reports can be specified:

phpbench run --report=aggregate --report=env

You can also tag runs and generate reports afterwards:

phpbench run --tag=main
phpbench report --ref=main --report=aggregate

For more information on storage see storage.

Configuring Reports

All reports can be configured either in the report configuration or directly on the command line using a simplified JSON encoded string instead of the report name:

To configure a report in phpbench.json:

{
    "runner.path": "NothingBench.php",
    "runner.executor": "debug",
    "report.generators": {
        "my-report": {
            "generator": "expression",
            "cols": [ "benchmark", "tag", "subject", "mean" ],
            "break": ["benchmark"]
        }
    }
}

Then run it with:

phpbench run --report=my-report

You can also configure reports directly from the command line using simplified JSON:

phpbench run --report='"generator": "expression", "cols": [ "benchmark", "tag", "subject", "mean" ], "break": ["benchmark"]'

In each case it is required to specify the generator key which corresponds to the registered name of the report generator.

You may also extend an existing report configuration:

phpbench run --report='extends: aggregate, break: ["benchmark", "set"]'

This will merge the given keys onto the configuration for the aggregate report.

Default Reports

aggregate

Shows aggregate details of each set of iterations.

Run:

phpbench run --report=aggregate

And get something like:

+--------------+--------------+-----+------+-----+----------+----------+--------+
| benchmark    | subject      | set | revs | its | mem_peak | mode     | rstdev |
+--------------+--------------+-----+------+-----+----------+----------+--------+
| NothingBench | benchNothing |     | 1    | 1   | 100b     | 10.000μs | ±0.00% |
+--------------+--------------+-----+------+-----+----------+----------+--------+

It is uses the table generator, see Expression Report for more information.

benchmark

Sumarize each benchmark:

Run:

phpbench run --report=benchmark --iterations=4

And get something like:

NothingBench
============

Average iteration times by variant

10.0μs    │ █ 
8.8μs     │ █ 
7.5μs     │ █ 
6.3μs     │ █ 
5.0μs     │ █ 
3.8μs     │ █ 
2.5μs     │ █ 
1.3μs     │ █ 
          └───
            1 

[█ <current>] 

1: benchNothing 

Memory by variant

100b      │ █ 
88b       │ █ 
75b       │ █ 
63b       │ █ 
50b       │ █ 
38b       │ █ 
25b       │ █ 
13b       │ █ 
          └───
            1 

[█ <current>] 

1: benchNothing 

<current>
+-----------------+--------+----------+--------+---------+
| subject         | memory | mode     | rstdev | stdev   |
+-----------------+--------+----------+--------+---------+
| benchNothing () | 100b   | 10.000μs | ±0.00% | 0.000μs |
+-----------------+--------+----------+--------+---------+

default

The default report presents the result of each iteration:

Run:

phpbench run --report=default --iterations=4

And get something like:

+------+--------------+--------------+-----+------+----------+----------+--------------+----------------+
| iter | benchmark    | subject      | set | revs | mem_peak | time_avg | comp_z_value | comp_deviation |
+------+--------------+--------------+-----+------+----------+----------+--------------+----------------+
| 0    | NothingBench | benchNothing |     | 1    | 100b     | 10.000μs | +0.00σ       | +0.00%         |
| 1    | NothingBench | benchNothing |     | 1    | 100b     | 10.000μs | +0.00σ       | +0.00%         |
| 2    | NothingBench | benchNothing |     | 1    | 100b     | 10.000μs | +0.00σ       | +0.00%         |
| 3    | NothingBench | benchNothing |     | 1    | 100b     | 10.000μs | +0.00σ       | +0.00%         |
+------+--------------+--------------+-----+------+----------+----------+--------------+----------------+

It is uses the table generator, see Expression Report for more information.

env

This report shows information about the environment that the benchmarks were executed in.

Run:

phpbench run --report=env

And get something like:

Suite E3X6AeMdP7L9E7E0X0A7McP1L8E1EdXbAbMbP7La xxxx-xx-xx xx-xx-xx
+----------+----------+-------+
| provider | key      | value |
+----------+----------+-------+
| test     | example1 | 1     |
| test     | example2 | 2     |
+----------+----------+-------+

Generator: Env.

Columns:

  • provider: Name of the environment provider (see PhpBench\Environment\Provider in the code for more information).
  • key: Information key.
  • value: Information value.

See the Environment reference for more information.

Note

The information available will differ depending on platform. For example, unix-sysload is only available on UNIX platforms, where as the VCS field will appear only when a supported VCS system is being used.

overview

Shows overview of a single or mutiple runs:

Run:

phpbench run --report=overview --iterations=4

And get something like:

Overview
========

Suites
+-----------+---------------------+------+------------+--------+------------+------+----------+----------+
| suite     | date                | php  | vcs branch | xdebug | iterations | revs | mode     | net_time |
+-----------+---------------------+------+------------+--------+------------+------+----------+----------+
| <current> | xxxx-xx-xx xx-xx-xx | null | null       | null   | 4          | 4    | 10.000μs | 40.000μs |
+-----------+---------------------+------+------------+--------+------------+------+----------+----------+

Average iteration times aggregated by benchmark

10.0μs    │ █ 
8.8μs     │ █ 
7.5μs     │ █ 
6.3μs     │ █ 
5.0μs     │ █ 
3.8μs     │ █ 
2.5μs     │ █ 
1.3μs     │ █ 
          └───
            1 

[█ <current>] 

1: NothingBench 

Average peak memory aggregated by benchmark

100b      │ █ 
88b       │ █ 
75b       │ █ 
63b       │ █ 
50b       │ █ 
38b       │ █ 
25b       │ █ 
13b       │ █ 
          └───
            1 

[█ <current>] 

1: NothingBench 

By Benchmark

10.0μs    │ █ 
8.8μs     │ █ 
7.5μs     │ █ 
6.3μs     │ █ 
5.0μs     │ █ 
3.8μs     │ █ 
2.5μs     │ █ 
1.3μs     │ █ 
          └───
            1 

[█ NothingBench] 

1: <current> 

Table
=====

<current>
+--------------+-----------------+--------+----------+----------+----------+--------+---------+
| benchmark    | subject         | memory | min      | max      | mode     | rstdev | stdev   |
+--------------+-----------------+--------+----------+----------+----------+--------+---------+
| NothingBench | benchNothing () | 100b   | 10.000μs | 10.000μs | 10.000μs | ±0.00% | 0.000μs |
+--------------+-----------------+--------+----------+----------+----------+--------+---------+