The benchmark runner is a command line application which executes the benchmarks and generates reports from the results.
To run all benchmarks in a specific directory:
$ phpbench run /path/to
To run a single benchmark class, specify a specific file:
$ phpbench run /path/to/HashBench.php
To run a single method of a single benchmark class, use the
$ phpbench run /path/to/HashBench.php --filter=benchMd5
Groups can be specified using the
$ phpbench run /path/to/HashBench.php --group=hash
--group options may be specified multiple
--filter option accepts a regex without the delimiters and matches
against a string such as
HashBench::benchMd5, so all of the following are
$ phpbench run /path/to --filter=benchFoo $ phpbench run /path/to --filter=HashBench::benchFoo $ phpbench run /path/to --filter=Hash.*
In addition the
--variant option allows you to filter by a specific
parameter set name:
$ phpbench run /path/to --filter=Hash.* --variant=md5
Overriding Iterations and Revolutions¶
$ phpbench run /path/to/HashBench.php --iterations=10 --revs=1000
You may specify these options multiple times.
Overriding the Bootstrap¶
You can override or set the runner.bootstrap using the
$ phpbench run /path/to/HashBench.php --bootstrap=vendor/autoload.php
Assertions: Overriding and Toleration¶
--assert option to introduce or override assertions:
$ phpbench run /path/to/HashBench.php --assert="variant.mode <= 10 microsconds +/- 10%"
This will assert that ALL variants must have a mode less than 10. For more information on assertions see Assertions.
Failing assertions will cause PHPBench to exit with code
2. If you want to
tolerate failures (f.e. in an unstable CI environment) you can use the
For more information about assertions see Assertions.
By default PHPBench will run the benchmarks and tell you that the benchmarks have been executed successfully. In order to see some useful information you can specify that a report be generated.
By default there are two reports
aggregate, and they can
be specified directly using the
$ phpbench run /path/to/HashBench.php --report=default
See the Reports guide for more information on how you can configure reports.
If you want to suppress all other output and only show the output from the
reports you can use the
--progress=none option. This is especially useful when
piping a report to another program.
PHPBench is able to significantly improve the stability of your benchmarks by retrying the iteration set until all the deviations in time between iterations fit within a given margin of error.
You can set this as follows:
$ phpbench run /path/to/HashBench.php --retry-threshold=5
The retry threshold is the margin of error as a percentage which is allowed between deviations. Generally the lower this value, the higher the stability, but the longer it will take for a set of iterations to be resolved.
By default the retry threshold is disabled.
You may also set the retry threshold in the configuration.
Changing the Output Format¶
By default PHPBench will output the reports to the console using the
console output. The output can be changed using the
For example, to render a HTML document:
$ phpbench run /path/to/HashBench.php --report=default --output=delimited
See the Report Renderers reference for more information.
You can store benchmark results locally:
$ phpbench run /path/to/HashBench.php --tag=foobar
Then generate reports:
$ phpbench report --ref=foobar --report=aggregate
To dump the benchmark results to an XML file use the
$ phpbench run /path/to/HashBench.php --dump-file=report.xml
Then generate reports:
$ phpbench report --file=report.xml --report=default
By default PHPBench issues a single
. for each benchmark subject executed.
This is the
dots progress reporter. Different progress reporters can be
specified using the
$ phpbench run /path/to/HashBench.php --progress=classdots
The built-in progress loggers are:
verbose: The default logger, format: [R<retry nb.>] I<iter nb.> P<parameter set nb.> <mean|mode per rev.> <standard deviation per rev.> <relative standard deviation per rev.> ).
plain: Similar to verbose, but without any animation, useful for CI environments.
dots: Shows one dot per subject (like PHPUnit).
classdots: Shows the benchmark class, and then a dot for each subject.
blinken: Highly visual progress logger.
histogram: Shows a histogram with 8 vertical levels and 16 bins for each iteration set.
All of the progress reports contain the following footer:
3 subjects, 30 iterations, 30000 revs, 0 rejects min [mean mode] max: 0.84 [1.13 1.12] 1.66 (μs/r) ⅀T: 33987μs μSD/r 0.16μs μRSD/r: 14.92%
It provides a summary of the minimum, mean, mode, and maximum subject times, given microseconds per revolution. ⅀T is the aggregate total time, μSD/r is the mean standard deviation, and μRSD/r is the mean relative standard deviation.
These summary statistics can be misleading. You should always verify the individual subject statistics before drawing any conclusions.
A custom configuration file can be specified with the
--config option. See
the Configuration reference for more information on configuration.
The following exit codes can occur:
0: Everything was fine.
1: Errors encountered in benchmarks.
2: Assertion failures.
255: Internal error