Benchmark Runner

The benchmark runner is a command line application which executes the benchmarks and generates reports from the results.

Running Benchmarks

To run all benchmarks in a specific directory:

$ phpbench run /path/to

To run a single benchmark class, specify a specific file:

$ phpbench run /path/to/HashBench.php

To run a single method of a single benchmark class, use the --filter option:

$ phpbench run /path/to/HashBench.php --filter=benchMd5

Groups can be specified using the --group option:

$ phpbench run /path/to/HashBench.php --group=hash

Note

Both --subject and --group options may be specified multiple times.

Filtering

The --filter option accepts a regex without the delimiters and matches against a string such as HashBench::benchMd5, so all of the following are valid:

$ phpbench run /path/to --filter=benchFoo
$ phpbench run /path/to --filter=HashBench::benchFoo
$ phpbench run /path/to --filter=Hash.*

In addition the --variant option allows you to filter by a specific parameter set name:

$ phpbench run /path/to --filter=Hash.* --variant=md5

Overriding Iterations and Revolutions

The benchmark runner can override the number of revolutions and iterations which will be executed:

$ phpbench run /path/to/HashBench.php --iterations=10 --revs=1000

You may specify these options multiple times.

Overriding the Bootstrap

You can override or set the runner.bootstrap using the --bootstrap option:

$ phpbench run /path/to/HashBench.php --bootstrap=vendor/autoload.php

Assertions: Overriding and Toleration

Use the --assert option to introduce or override assertions:

$ phpbench run /path/to/HashBench.php --assert="variant.mode <= 10 microsconds +/- 10%"

This will assert that ALL variants must have a mode less than 10. For more information on assertions see Assertions.

Failing assertions will cause PHPBench to exit with code 2. If you want to tolerate failures (f.e. in an unstable CI environment) you can use the --tolerate-failure option.

For more information about assertions see Assertions.

Generating Reports

By default PHPBench will run the benchmarks and tell you that the benchmarks have been executed successfully. In order to see some useful information you can specify that a report be generated.

By default there are two reports default and aggregate, and they can be specified directly using the --report option:

$ phpbench run /path/to/HashBench.php --report=default

See the Reports guide for more information on how you can configure reports.

Note

If you want to suppress all other output and only show the output from the reports you can use the --progress=none option. This is especially useful when piping a report to another program.

Retry Threshold

PHPBench is able to significantly improve the stability of your benchmarks by retrying the iteration set until all the deviations in time between iterations fit within a given margin of error.

You can set this as follows:

$ phpbench run /path/to/HashBench.php --retry-threshold=5

The retry threshold is the margin of error as a percentage which is allowed between deviations. Generally the lower this value, the higher the stability, but the longer it will take for a set of iterations to be resolved.

By default the retry threshold is disabled.

You may also set the retry threshold in the configuration.

Changing the Output Format

By default PHPBench will output the reports to the console using the console output. The output can be changed using the --output option. For example, to render a HTML document:

$ phpbench run /path/to/HashBench.php --report=default --output=delimited

See the Report Renderers reference for more information.

Storing Results

You can store benchmark results locally:

$ phpbench run /path/to/HashBench.php --tag=foobar

Then generate reports:

$ phpbench report --ref=foobar --report=aggregate

To dump the benchmark results to an XML file use the --dump-file option:

$ phpbench run /path/to/HashBench.php --dump-file=report.xml

Then generate reports:

$ phpbench report --file=report.xml --report=default

Progress Reporters

By default PHPBench issues a single . for each benchmark subject executed. This is the dots progress reporter. Different progress reporters can be specified using the --progress option:

../_images/blinken.gif

blinken progress logger.

$ phpbench run /path/to/HashBench.php --progress=classdots

The built-in progress loggers are:

  • verbose: The default logger, format: [R<retry nb.>] I<iter nb.> P<parameter set nb.> <mean|mode per rev.> <standard deviation per rev.> <relative standard deviation per rev.> ).

  • plain: Similar to verbose, but without any animation, useful for CI environments.

  • dots: Shows one dot per subject (like PHPUnit).

  • classdots: Shows the benchmark class, and then a dot for each subject.

  • blinken: Highly visual progress logger.

  • histogram: Shows a histogram with 8 vertical levels and 16 bins for each iteration set.

All of the progress reports contain the following footer:

3 subjects, 30 iterations, 30000 revs, 0 rejects
min [mean mode] max: 0.84 [1.13 1.12] 1.66 (μs/r)
⅀T: 33987μs μSD/r 0.16μs μRSD/r: 14.92%

It provides a summary of the minimum, mean, mode, and maximum subject times, given microseconds per revolution. ⅀T is the aggregate total time, μSD/r is the mean standard deviation, and μRSD/r is the mean relative standard deviation.

Warning

These summary statistics can be misleading. You should always verify the individual subject statistics before drawing any conclusions.

Configuration File

A custom configuration file can be specified with the --config option. See the Configuration reference for more information on configuration.

Exit codes

The following exit codes can occur:

  • 0: Everything was fine.

  • 1: Errors encountered in benchmarks.

  • 2: Assertion failures.

  • 255: Internal error