Writing Benchmarks

Benchmark classes have the following characteristics:

  • The class and filename must be the same.
  • Class methods that start with bench will be executed by the benchrunner and timed.

PHPBench does not require that the benchmark class be aware of PHPBench library - it does not need to extend a parent class or implement an interface.

The following is a simple benchmark class:

<?php
// HashBench.php

class HashBench
{
    public function benchMd5()
    {
        hash('md5', 'Hello World!');
    }

    public function benchSha1()
    {
        hash('sha1', 'Hello World!');
    }
}

And it can be executed as follows:

$ phpbench run examples/HashBench.php --progress=dots
PhpBench 0.8.0-dev. Running benchmarks.

...

3 subjects, 30 iterations, 30000 revs, 0 rejects
⅀T: 30543μs μSD/r 0.05μs μRSD/r: 4.83%
min mean max: 0.78 1.02 1.47 (μs/r)

Note

The above command does not generate a report, add --report=default to view something useful.

PHPBench reads docblock annotations in the benchmark class. Annotations can be placed in the class docblock, or on individual methods docblocks.

Note

Instead of prefixing a method with bench you can use the @Subject annotation or specify a custom pattern.

Improving Precision: Revolutions

When testing units of code where microsecond accuracy is important, it is necessary to increase the number of revolutions performed by the benchmark runner. The term “revolutions” (invented here) refers to the number of times the benchmark is executed consecutively within a single time measurement.

We can arrive at a more accurate measurement by determining the mean time from multiple revolutions (i.e. time / revolutions) than we could with a single revolution. In other words, more revolutions means more precision.

Revolutions can be specified using the @Revs annotation:

<?php

/**
 * @Revs(1000)
 */
class HashBench
{
    // ...
}

You may also specify an array:

<?php

/**
 * @Revs({1, 8, 64, 4096})
 */
class HashBench
{
    // ...
}

Revolutions can also be overridden from the command line.

Verifying and Improving Stability: Iterations

Iterations represent the number of times we will perform the benchmark (including all the revolutions). Contrary to revolutions, a time reading will be taken for each iteration.

By looking at the separate time measurement of each iteration we can determine how stable the readings are. The less the measurements differ from each other, the more stable the benchmark is, and the more you can trust the results.

Note

In a perfect environment the readings would all be exactly the same - but such an environment is unlikely to exist

Iterations can be specified using the @Iterations annotation:

<?php

/**
 * @Iterations(5)
 */
class HashBench
{
    // ...
}

As with revolutions, you may also specify an array.

Iterations can also be overridden from the command line.

You can instruct PHPBench to continuously run the iterations until the deviation of each iteration fits within a given margin of error by using the --retry-threshold. See Retry Threshold for more information.

Subject (runtime) State: Before and After

Any number of methods can be executed both before and after each benchmark subject using the @BeforeMethods and @AfterMethods annotations. Before methods are useful for bootstrapping your environment, for example:

<?php

/**
 * @BeforeMethods({"init"})
 */
class HashBench
{
    private $hasher;

    public function init()
    {
        $this->hasher = new Hasher();
    }

    public function benchMd5()
    {
        $this->hasher->md5('Hello World!');
    }
}

Multiple before and after methods can be specified.

Note

If before and after methods are used when the @ParamProviders annotations are used, then they will also be passed the parameters.

Benchmark (external) State: Before and After

Sometimes you will want to perform actions which establish an external state. For example, creating or populating a database, creating files, etc.

This can be achieved by creating static methods within your benchmark class and adding the @BeforeClassMethods and @AfterClassMethods:

These methods will be executed by the runner once per benchmark class.

<?php

/**
 * @BeforeClassMethods({"initDatabase"})
 */
class DatabaseBench
{
    public static function initDatabase()
    {
        // init database here.
    }

    // ...
}

Note

These methods are static and are executed in a process that is separate from that from which your iterations will be executed. Therefore state will not be carried over to your iterations!.

Parameterized Benchmarks

Parameter sets can be provided to benchmark subjects. For example:

<?php

class HashBench
{
    public function provideStrings()
    {
        yield 'hello' => [ 'string' => 'Hello World!' ];
        yield 'goodbye' => [ 'string' => 'Goodbye Cruel World!' ];
    }

    /**
     * @ParamProviders({"provideStrings"})
     */
    public function benchMd5($params)
    {
        hash('md5', $params['string']);
    }
}

The benchMd5 subject will now be benchmarked with each parameter set.

The param provider can return a set of parameters using any iterable. For example the above could also be retuned as an array:

<?php

class HashBench
{
    public function provideStrings()
    {
        return [
            'hello' => [ 'string' => 'Hello World!' ],
            'goodbye' => [ 'string' => 'Goodbye Cruel World!' ]
        ];
    }
}

Warning

It should be noted that Generators are consumed completely before the subject is executed. If you have a very large data set, it will be read completely into memory.

Multiple parameter providers can be used, in which case the data sets will be combined into a cartesian product - all possible combinations of the parameters will be generated, for example:

<?php

class HashBench
{
    public function provideStrings()
    {
        yield 'hello' => [ 'string' => 'Hello World!' ];
        yield 'goodbye' => [ 'string' => 'Goodbye Cruel World!' ];
    }

    public function provideNumbers()
    {
        yield 'md5' => [ 'algorithm' => 'md5' ];
        yield 'sha1' => [ 'algorithm' => 'sha1' ];
    }

    /**
     * @ParamProviders({"provideStrings", "provideNumbers"})
     */
    public function benchHash($params)
    {
        hash($params['algorithm'], $params['string']);
    }
}

Will result in the following parameter benchmark scenarios:

<?php

// #0
['string' => 'Hello World!', 'algorithm' => 'md5'];

// #1
['string' => 'Goodbye Cruel World!', 'algorithm' => 'md5'[;

// #2
['string' => 'Hello World!', 'algorithm' => 'sha1'];

// #3
['string' => 'Goodbye Cruel World!', 'algorithm' => 'sha1'];

Groups

You can assign benchmark subjects to groups using the @Groups annotation.

<?php

/**
 * @Groups({"hash"})
 */
class HashBench
{
    // ...
}

The group can then be targeted using the command line interface.

Skipping Subjects

You can skip subjects by using the @Skip annotation:

<?php

class HashBench extends Foobar
{
    /**
     * @Skip()
     */
    public function testFoobar()
    {
    }
}

Extending Existing Array Values

When working with annotations which accept an array value, you may wish to extend the values of the same annotation from ancestor classes. This can be accomplished using the extend option.

<?php

abstract class AbstractHash
{
    /**
     * @Groups({"md5"})
     */
    abstract public function benchMd5();
}

/**
 * @Groups({"my_hash_implementation"}, extend=true)
 */
class HashBench extends AbstractHash
{
    public function benchMd5()
    {
        // ...
    }
}

The benchHash subject will now be in both the md5 and my_hash_implementation groups.

This option is available on all array valued (plural) annotations.

Recovery Period: Sleeping

Sometimes it may be necessary to pause between iterations in order to let the system recover. Use the @Sleep annotation, specifying the number of microseconds required:

<?php

class HashBench
{
    /**
     * @Iterations(10)
     * @Sleep(1000000)
     */
    public function benchMd5()
    {
        md5('Hello World');
    }
}

The above example will pause (sleep) for 1 second after each iteration.

Note

This can be overridden using the --sleep option from the CLI.

Microseconds to Minutes: Time Units

If you have benchmarks which take seconds or even minutes to execute then the default time unit, microseconds, is going to be far more visual precision than you need and will only serve to make the results more difficult to interpret.

You can specify output time units using the @OutputTimeUnit annotation (precision is optional):

<?php

class HashBench
{
    /**
     * @Iterations(10)
       @OutputTimeUnit("seconds", precision=3)
     */
    public function benchSleep()
    {
        sleep(2);
    }
}

The following time units are available:

  • microseconds
  • milliseconds
  • seconds
  • minutes
  • hours
  • days

Mode: Throughput Representation

The output mode determines how the measurements are presented, either time or throughput. time mode is the default and shows the average execution time of a single revolution. throughput shows how many operations are executed within a single time unit:

<?php

class HashBench
{
    /**
     * @OutputTimeUnit("seconds")
     * @OutputMode("throughput")
     */
    public function benchMd5()
    {
        hash('md5', 'Hello World!');
    }
}

PHPBench will then render all measurements for benchMd5 similar to 363,874.536ops/s.

Warming Up: Getting ready for the show

In some cases, it might be a good idea to execute a revolution or two before performing the revolutions time measurement.

For example, when benchmarking something that uses an class autoloader, the first revolution will always be slower because the autoloader will not to be called again.

Use the @Warmup annotation to execute any number of revolutions before actually measuring the revolutions time.

<?php

// ...
class ReportBench
{
    // ...

    /**
     * @Warmup(2)
     * @Revs(10)
     */
    public function benchGenerateReport()
    {
        $this->generator->generateMyComplexReport();
    }
}

As with revolutions, you may also specify an array.

Timeout: Bailing when things take too long

Use the @Timeout annotation to specify the maximum number of seconds before an iteration timesout and fails. The following example will fail after 0.1 seconds:

<?php

// ...
class ReportBench
{
    /**
     * @Timeout(0.1)
     */
    public function benchGenerateReport()
    {
       sleep(1);
    }
}

Assertions

Warning

Assertions are absolute, benchmarks are relative to the environment they are running in.

If you use them in a continuous integration environment the stability of your build will depend on the state of the environment, you can prevent failing builds with the –tolerate-failure option.

Assertions allow you to specify what a valid range is for a given statistic, for example, “the mean must be less than 10”.

<?php

// ...
class AssertiveBench
{
    // ...

    /**
     * @Assert(stat="mean", value="10")
     */
    public function benchGenerateReport()
    {
        // ...
    }
}

By default the comparator is < (less than), you can also specify > using the comparator key:

<?php

class AssertiveBench
{
    // ...

    /**
     * @Assert(stat="mean", value="10", comparator=">")
     */
    public function benchGenerateReport()
    {
        // ...
    }
}

The default time unit for assertions is microseconds, but you can specify any supported time unit and you can also change the mode to throughput:

<?php

class AssertiveBench
{
    // ...

    /**
     * @Assert(stat="mean", value="10", comparator=">", time_unit="milliseconds", mode="throughput")
     */
    public function benchGenerateReport()
    {
        // ...
    }
}

The above will assert that an average of more than 10 operations are completed in a millisecond. See Microseconds to Minutes: Time Units and Mode: Throughput Representation for more information.

For more information about assertions see Asserters.