Expression Report¶
Note
For custom reports it is now recommended to use the Component generator.
The expression generator is the generator that allows you to analyze your benchmarking results. It uses the PHPBench expression language to evaluate tabular data:
phpbench run --report=expression --executor=debug NothingBench.php --progress=none
Yields something like:
+-----------+--------------+--------------+-----+------+-----+----------+----------+----------+----------+----------+---------+--------+
| tag | benchmark | subject | set | revs | its | mem_peak | best | mode | mean | worst | stdev | rstdev |
+-----------+--------------+--------------+-----+------+-----+----------+----------+----------+----------+----------+---------+--------+
| <current> | NothingBench | benchNothing | | 1 | 1 | 100b | 10.000μs | 10.000μs | 10.000μs | 10.000μs | 0.000μs | ±0.00% |
+-----------+--------------+--------------+-----+------+-----+----------+----------+----------+----------+----------+---------+--------+
Options¶
- title:
Type(s):
[null, string]
, Default:NULL
Title to use for report
- description:
Type(s):
[null, string]
, Default:NULL
Description to use for report
- cols:
Type(s):
[array, null]
, Default:NULL
Columns to display
- expressions:
Type(s):
array
, Default:[]
Map from column names to expressions
- baseline_expressions:
Type(s):
array
, Default:[]
When the baseline is used, expressions here will be merged with the
expressions
.
- aggregate:
Type(s):
array
, Default:[suite_tag, benchmark_class, subject_name, variant_index]
Group rows by these columns
- break:
Type(s):
array
, Default:[]
Group tables by these columns
- include_baseline:
Type(s):
bool
, Default:false
If the baseline should be included as additional rows, or if it should be inlined
Columns¶
The visible columns are dictated by the cols
configuration:
{
"report.generators": {
"my-report": {
"generator": "expression",
"cols": ["subject", "mode"]
}
}
}
When using the report:
phpbench run --report=my-report --executor=debug NothingBench.php --progress=none
It will only show the selected columns:
+--------------+----------+
| subject | mode |
+--------------+----------+
| benchNothing | 10.000μs |
+--------------+----------+
You can also override expressions by passing a map:
{
"report.generators": {
"my-report": {
"generator": "expression",
"cols": {
"subject": null,
"mode": null,
"hello": "format(\"Hello World: %s\", \"Foobar\")"
}
}
}
}
Which yields:
+--------------+----------+---------------------+
| subject | mode | hello |
+--------------+----------+---------------------+
| benchNothing | 10.000μs | Hello World: Foobar |
+--------------+----------+---------------------+
Aggregate¶
Aggregation decides which values are included in each row - should each row contain only the values for a single iteration? should all values for the variant by included? should we include all values for the entire suite? (not recommended).
{
"report.generators": {
"my-report": {
"generator": "expression",
"aggregate": ["benchmark_class", "subject_name", "variant_name", "iteration_index"]
}
}
}
This will aggregate by unique values of the named columns, producing a single row per iteration:
+-----------+--------------+--------------+-----+------+-----+----------+----------+----------+----------+----------+---------+--------+
| tag | benchmark | subject | set | revs | its | mem_peak | best | mode | mean | worst | stdev | rstdev |
+-----------+--------------+--------------+-----+------+-----+----------+----------+----------+----------+----------+---------+--------+
| <current> | NothingBench | benchNothing | | 1 | 5 | 100b | 10.000μs | 10.000μs | 10.000μs | 10.000μs | 0.000μs | ±0.00% |
| <current> | NothingBench | benchNothing | | 1 | 5 | 100b | 10.000μs | 10.000μs | 10.000μs | 10.000μs | 0.000μs | ±0.00% |
| <current> | NothingBench | benchNothing | | 1 | 5 | 100b | 10.000μs | 10.000μs | 10.000μs | 10.000μs | 0.000μs | ±0.00% |
| <current> | NothingBench | benchNothing | | 1 | 5 | 100b | 10.000μs | 10.000μs | 10.000μs | 10.000μs | 0.000μs | ±0.00% |
| <current> | NothingBench | benchNothing | | 1 | 5 | 100b | 10.000μs | 10.000μs | 10.000μs | 10.000μs | 0.000μs | ±0.00% |
+-----------+--------------+--------------+-----+------+-----+----------+----------+----------+----------+----------+---------+--------+
Break¶
You can partition the report into multiple tables by using the break
option:
{
"report.generators": {
"my-report": {
"generator": "expression",
"break": ["benchmark"],
"cols": ["benchmark","subject", "set", "revs", "its", "mem_peak", "mode", "rstdev"]
}
}
}
Now each benchmark class will get its own table:
MultipleSubjectBench
+---------------+-----+------+-----+----------+----------+--------+
| subject | set | revs | its | mem_peak | mode | rstdev |
+---------------+-----+------+-----+----------+----------+--------+
| benchSubject1 | | 1 | 1 | 100b | 10.000μs | ±0.00% |
| benchSubject2 | | 1 | 1 | 100b | 10.000μs | ±0.00% |
| benchSubject3 | | 1 | 1 | 100b | 10.000μs | ±0.00% |
+---------------+-----+------+-----+----------+----------+--------+
NothingBench
+--------------+-----+------+-----+----------+----------+--------+
| subject | set | revs | its | mem_peak | mode | rstdev |
+--------------+-----+------+-----+----------+----------+--------+
| benchNothing | | 1 | 1 | 100b | 10.000μs | ±0.00% |
+--------------+-----+------+-----+----------+----------+--------+
Expressions¶
The expressions define the available columns, you can add or override expressions:
{
"report.generators": {
"my-report": {
"generator": "expression",
"expressions": {
"mode": "\"This is the mode: \" ~ mode(result_time_avg)"
},
"cols": [ "benchmark", "subject", "mode" ]
}
}
}
Which yields:
+--------------+--------------+----------------------+
| benchmark | subject | mode |
+--------------+--------------+----------------------+
| NothingBench | benchNothing | This is the mode: 10 |
+--------------+--------------+----------------------+
Data¶
The expressions have access to all aggregated data, and in addition, the
entire result set via. the suite
variable.
The Aggregated data is provided as an array of column names to values:
{
// ...
"subject_name": ["benchFoobar", "benchFoobar", "benchFoobar"],
"result_time_net": [10, 20, 30],
// ...
}
So the mode
for result_time_net
could be calculated via the expression
mode(result_time_net)
.
The suite
variable is data frame that represents the entire result set and
can be used to access a specific value through filtering. In the contrived example below we calculate the difference
between the mode of a referenced subject against that of the current variant:
{
"report.generators": {
"my-report": {
"generator": "expression",
"cols": {
"subject": null,
"difference": "percent_diff(mode(result_time_avg), mode(suite[subject_name = \"benchNothing\"][\"result_time_avg\"]))"
}
}
}
}
Yielding:
+--------------+------------+
| subject | difference |
+--------------+------------+
| benchNothing | 0.00% |
+--------------+------------+
You can get a list of all available columns with:
phpbench run --report='extends:bare,vertical:true' --executor=debug NothingBench.php --progress=none
Yielding:
+------------------------+---------------+
| field | value |
+------------------------+---------------+
| has_baseline | false |
| benchmark_name | NothingBench |
| benchmark_class | \NothingBench |
| subject_name | benchNothing |
| subject_groups | [] |
| subject_time_unit | null |
| subject_time_precision | null |
| subject_time_mode | null |
| variant_index | 0 |
| variant_name | |
| variant_params | [] |
| variant_revs | 1 |
| variant_iterations | 1 |
| suite_tag | <current> |
| suite_date | xxxx-xx-xx |
| suite_time | xx-xx-xx |
| iteration_index | 0 |
| env_test_example1 | 1 |
| env_test_example2 | 2 |
| result_mem_peak | 100 |
| result_mem_real | 100 |
| result_mem_final | 100 |
| result_time_net | 10 |
| result_time_revs | 1 |
| result_time_avg | 10 |
| result_comp_z_value | 0 |
| result_comp_deviation | 0 |
+------------------------+---------------+
Note that any additional result and environment data will also be included in the form result_<type>_<metric> and env_<type>_<metric>.