Expression Report

Note

For custom reports it is now recommended to use the Component generator.

The expression generator is the generator that allows you to analyze your benchmarking results. It uses the PHPBench expression language to evaluate tabular data:

phpbench run --report=expression --executor=debug NothingBench.php --progress=none

Yields something like:

+-----------+--------------+--------------+-----+------+-----+----------+----------+----------+----------+----------+---------+--------+
| tag       | benchmark    | subject      | set | revs | its | mem_peak | best     | mode     | mean     | worst    | stdev   | rstdev |
+-----------+--------------+--------------+-----+------+-----+----------+----------+----------+----------+----------+---------+--------+
| <current> | NothingBench | benchNothing |     | 1    | 1   | 100b     | 10.000μs | 10.000μs | 10.000μs | 10.000μs | 0.000μs | ±0.00% |
+-----------+--------------+--------------+-----+------+-----+----------+----------+----------+----------+----------+---------+--------+

Options

title:

Type(s): [null, string], Default: NULL

Title to use for report

description:

Type(s): [null, string], Default: NULL

Description to use for report

cols:

Type(s): [array, null], Default: NULL

Columns to display

expressions:

Type(s): array, Default: []

Map from column names to expressions

baseline_expressions:

Type(s): array, Default: []

When the baseline is used, expressions here will be merged with the expressions.

aggregate:

Type(s): array, Default: [suite_tag, benchmark_class, subject_name, variant_index]

Group rows by these columns

break:

Type(s): array, Default: []

Group tables by these columns

include_baseline:

Type(s): bool, Default: false

If the baseline should be included as additional rows, or if it should be inlined

Columns

The visible columns are dictated by the cols configuration:

{
    "report.generators": {
         "my-report": {
             "generator": "expression",
             "cols": ["subject", "mode"]
         }
    }
}

When using the report:

phpbench run --report=my-report --executor=debug NothingBench.php --progress=none

It will only show the selected columns:

+--------------+----------+
| subject      | mode     |
+--------------+----------+
| benchNothing | 10.000μs |
+--------------+----------+

You can also override expressions by passing a map:

{
    "report.generators": {
         "my-report": {
             "generator": "expression",
             "cols": {
                 "subject": null,
                 "mode": null,
                 "hello": "format(\"Hello World: %s\", \"Foobar\")"
             }
         }
    }
}

Which yields:

+--------------+----------+---------------------+
| subject      | mode     | hello               |
+--------------+----------+---------------------+
| benchNothing | 10.000μs | Hello World: Foobar |
+--------------+----------+---------------------+

For example you add a diff column:

{
    "runner.executor": "debug-example",
    "runner.iterations": 5,
    "runner.executors": {
        "debug-example": {
            "executor": "debug",
            "times":[10, 20, 30, 40, 50]
        }
    },
    "report.generators": {
         "my-report": {
             "generator": "expression",
             "aggregate": ["iteration_index"],
             "cols": {
                 "subject": null,
                 "mode": null,
                 "diff": "format('%.2fx', mean(result_time_avg) / min(suite['result_time_avg']))"
             }
         }
    }
}

Which yields:

+--------------+----------+-------+
| subject      | mode     | diff  |
+--------------+----------+-------+
| benchNothing | 10.000μs | 1.00x |
| benchNothing | 20.000μs | 2.00x |
| benchNothing | 30.000μs | 3.00x |
| benchNothing | 40.000μs | 4.00x |
| benchNothing | 50.000μs | 5.00x |
+--------------+----------+-------+

Aggregate

Aggregation decides which values are included in each row - should each row contain only the values for a single iteration? should all values for the variant by included? should we include all values for the entire suite? (not recommended).

{
    "report.generators": {
         "my-report": {
             "generator": "expression",
             "aggregate": ["benchmark_class", "subject_name", "variant_name", "iteration_index"]
         }
    }
}

This will aggregate by unique values of the named columns, producing a single row per iteration:

+-----------+--------------+--------------+-----+------+-----+----------+----------+----------+----------+----------+---------+--------+
| tag       | benchmark    | subject      | set | revs | its | mem_peak | best     | mode     | mean     | worst    | stdev   | rstdev |
+-----------+--------------+--------------+-----+------+-----+----------+----------+----------+----------+----------+---------+--------+
| <current> | NothingBench | benchNothing |     | 1    | 5   | 100b     | 10.000μs | 10.000μs | 10.000μs | 10.000μs | 0.000μs | ±0.00% |
| <current> | NothingBench | benchNothing |     | 1    | 5   | 100b     | 10.000μs | 10.000μs | 10.000μs | 10.000μs | 0.000μs | ±0.00% |
| <current> | NothingBench | benchNothing |     | 1    | 5   | 100b     | 10.000μs | 10.000μs | 10.000μs | 10.000μs | 0.000μs | ±0.00% |
| <current> | NothingBench | benchNothing |     | 1    | 5   | 100b     | 10.000μs | 10.000μs | 10.000μs | 10.000μs | 0.000μs | ±0.00% |
| <current> | NothingBench | benchNothing |     | 1    | 5   | 100b     | 10.000μs | 10.000μs | 10.000μs | 10.000μs | 0.000μs | ±0.00% |
+-----------+--------------+--------------+-----+------+-----+----------+----------+----------+----------+----------+---------+--------+

Break

You can partition the report into multiple tables by using the break option:

{
    "report.generators": {
         "my-report": {
             "generator": "expression",
             "break": ["benchmark"],
             "cols": ["benchmark","subject", "set", "revs", "its", "mem_peak", "mode", "rstdev"]
         }
    }
}

Now each benchmark class will get its own table:

MultipleSubjectBench
+---------------+-----+------+-----+----------+----------+--------+
| subject       | set | revs | its | mem_peak | mode     | rstdev |
+---------------+-----+------+-----+----------+----------+--------+
| benchSubject1 |     | 1    | 1   | 100b     | 10.000μs | ±0.00% |
| benchSubject2 |     | 1    | 1   | 100b     | 10.000μs | ±0.00% |
| benchSubject3 |     | 1    | 1   | 100b     | 10.000μs | ±0.00% |
+---------------+-----+------+-----+----------+----------+--------+

NothingBench
+--------------+-----+------+-----+----------+----------+--------+
| subject      | set | revs | its | mem_peak | mode     | rstdev |
+--------------+-----+------+-----+----------+----------+--------+
| benchNothing |     | 1    | 1   | 100b     | 10.000μs | ±0.00% |
+--------------+-----+------+-----+----------+----------+--------+

Expressions

The expressions define the available columns, you can add or override expressions:

{
    "report.generators": {
         "my-report": {
             "generator": "expression",
             "expressions": {
                "mode": "\"This is the mode: \" ~ mode(result_time_avg)"
             },
             "cols": [ "benchmark", "subject", "mode" ]
         }
    }
}

Which yields:

+--------------+--------------+----------------------+
| benchmark    | subject      | mode                 |
+--------------+--------------+----------------------+
| NothingBench | benchNothing | This is the mode: 10 |
+--------------+--------------+----------------------+

Data

The expressions have access to all aggregated data, and in addition, the entire result set via. the suite variable.

The Aggregated data is provided as an array of column names to values:

{
    // ...
    "subject_name": ["benchFoobar", "benchFoobar", "benchFoobar"],
    "result_time_net": [10, 20, 30],
    // ...
}

So the mode for result_time_net could be calculated via the expression mode(result_time_net).

The suite variable is data frame that represents the entire result set and can be used to access a specific value through filtering. In the contrived example below we calculate the difference between the mode of a referenced subject against that of the current variant:

{
    "report.generators": {
         "my-report": {
             "generator": "expression",
             "cols": {
                 "subject": null,
                 "difference": "percent_diff(mode(result_time_avg), mode(suite[subject_name = \"benchNothing\"][\"result_time_avg\"]))"
             }
         }
    }
}

Yielding:

+--------------+------------+
| subject      | difference |
+--------------+------------+
| benchNothing | 0.00%      |
+--------------+------------+

You can get a list of all available columns with:

phpbench run --report='extends:bare,vertical:true' --executor=debug NothingBench.php --progress=none

Yielding:

+------------------------+---------------+
| field                  | value         |
+------------------------+---------------+
| has_baseline           | false         |
| benchmark_name         | NothingBench  |
| benchmark_class        | \NothingBench |
| subject_name           | benchNothing  |
| subject_groups         | []            |
| subject_time_unit      | null          |
| subject_time_precision | null          |
| subject_time_mode      | null          |
| variant_index          | 0             |
| variant_name           |               |
| variant_params         | []            |
| variant_revs           | 1             |
| variant_iterations     | 1             |
| suite_tag              | <current>     |
| suite_date             | xxxx-xx-xx    |
| suite_time             | xx-xx-xx      |
| iteration_index        | 0             |
| env_test_example1      | 1             |
| env_test_example2      | 2             |
| result_mem_peak        | 100           |
| result_mem_real        | 100           |
| result_mem_final       | 100           |
| result_time_net        | 10            |
| result_time_revs       | 1             |
| result_time_avg        | 10            |
| result_comp_z_value    | 0             |
| result_comp_deviation  | 0             |
+------------------------+---------------+

Note that any additional result and environment data will also be included in the form result_<type>_<metric> and env_<type>_<metric>.