Annotributes

Configure your benchmarks with Annotations or, if you have PHP 8, Attributes. Note that some of these settings (e.g. runner.revs, runner.iterations) can be set globaly in the Configuration.

Revolutions

When testing units of code where microsecond accuracy is important, it is necessary to increase the number of revolutions performed by the benchmark runner. The term “revolutions” (invented here) refers to the number of times the benchmark is executed consecutively within a single time measurement.

We can arrive at a more accurate measurement by determining the mean time from multiple revolutions (i.e. time / revolutions) than we could with a single revolution. In other words, more revolutions means more precision.

Revolutions can be specified using the @Revs annotation:

/**
 */
class AnnotatedBench
{
    /**
     * @Revs(10)
     */
    public function benchTimeItself(): void
    {
        usleep(50);
    }

}
use PhpBench\Attributes as Bench;

class AttributedBench
{
    #[Bench\Revs(10)]
    public function benchTimeItself(): void
    {
        usleep(50);
    }

}

You may also specify an array:

/**
 * @Revs({1, 8, 64, 4096})
 */
class HashBench
{
    // ...
}

Revolutions can also be overridden from the command line.

Iterations

Iterations specify how many samples should be taken - i.e. how many times we run the revolutions and capture time and memory information (for example).

By looking at the separate time measurement of each iteration we can determine how stable the readings are. The less the measurements differ from each other, the more stable the benchmark.

Note

In a perfect environment the readings would all be exactly the same - but such an environment is unlikely to exist

Iterations can be specified using the @Iterations annotation:

/**
 */
class AnnotatedBench
{
    /**
     * @Iterations(10)
     */
    public function benchTimeItself(): void
    {
        usleep(50);
    }

}
use PhpBench\Attributes as Bench;

class AttributedBench
{
    #[Bench\Iterations(10)]
    public function benchTimeItself(): void
    {
        usleep(50);
    }

}

As with revolutions, you may also specify an array.

Iterations can also be overridden from the command line.

You can instruct PHPBench to continuously run the iterations until the deviation of each iteration fits within a given margin of error by using the --retry-threshold. See RetryThreshold for more information.

Benchmark Hooks

Method hooks

Any number of methods can be executed both before and after each benchmark subject using the @BeforeMethods and @AfterMethods annotations. Before methods are useful for bootstrapping your environment:

/**
 */
class AnnotatedBench
{
    /**
     * @BeforeMethods("setUp")
     * @AfterMethods("tearDown")
     */
    public function benchTimeItself(): void
    {
        usleep(50);
    }

    public function setUp(): void
    {
        // do somrthing before the benchmark
    }

    public function tearDown(): void
    {
        // do somrthing after the benchmark
    }

}
use PhpBench\Attributes as Bench;

class AttributedBench
{
    #[Bench\BeforeMethods('setUp')]
    #[Bench\AfterMethods("tearDown")]
    public function benchTimeItself(): void
    {
        usleep(50);
    }

    public function setUp(): void
    {
        // do somrthing before the benchmark
    }

    public function tearDown(): void
    {
        // do somrthing after the benchmark
    }

}

Multiple before and after methods can be specified.

Note

If before and after methods are used when the @ParamProviders annotations are used, then they will also be passed the parameters.

Class Hooks

Sometimes you will want to perform actions which establish an external state. For example, creating or populating a database, creating files, etc.

This can be achieved by creating static methods within your benchmark class and adding the @BeforeClassMethods and @AfterClassMethods:

These methods will be executed by the runner once per benchmark class.

/**
 * @BeforeClassMethods("setUpBeforeClass")
 * @AfterClassMethods("tearDownAfterClass")
 */
class AnnotatedBench
{
    /**
     */
    public function benchTimeItself(): void
    {
        usleep(50);
    }

    public static function setUpBeforeClass(): void
    {
        // do somrthing before the benchmark
    }

    public static function tearDownAfterClass(): void
    {
        // do somrthing after the benchmark
    }

}
use PhpBench\Attributes as Bench;

#[Bench\BeforeClassMethods(['setUpBeforeClass'])]
#[Bench\AfterClassMethodsClassMethods(['tearDownBeforeClass'])]
class AttributedBench
{
    public function benchTimeItself(): void
    {
        usleep(50);
    }

    public static function setUpBeforeClass(): void
    {
        // do somrthing before the benchmark
    }

    public static function tearDownAfterClass(): void
    {
        // do somrthing after the benchmark
    }

}

Note

These methods are static and are executed in a process that is separate from that from which your iterations will be executed. Therefore state will not be carried over to your iterations!.

Parameterized Benchmarks

Parameter sets can be provided to benchmark subjects:

/**
 */
class AnnotatedBench
{
    /**
     * @ParamProviders("provideMd5")
     */
    public function benchMd5(array $params): void
    {
        hash('md5', $params['string']);
    }

    public function provideMd5(): Generator
    {
        yield 'hello' => [ 'string' => 'Hello World!' ];
        yield 'goodbye' => [ 'string' => 'Goodbye Cruel World!' ];
    }

}
use PhpBench\Attributes as Bench;

class AttributedBench
{
    #[Bench\ParamProviders(['provideMd5'])]
    public function benchMd5(array $params): void
    {
        hash('md5', $params['string']);
    }

    public function provideMd5(): Generator
    {
        yield 'hello' => [ 'string' => 'Hello World!' ];
        yield 'goodbye' => [ 'string' => 'Goodbye Cruel World!' ];
    }

}

The benchMd5 subject will now be benchmarked with each parameter set.

The param provider can return a set of parameters using any iterable. For example the above could also be returned as an array:

/**
 */
class AnnotatedBench
{
    /**
     * @ParamProviders({"provideStringsAsArray"})
     */
    public function benchIterable(array $params): void
    {
        $helloThenGoodbye = $params['string'];
    }

    public function provideStringsAsArray(): array
    {
        return [
            'hello' => [ 'string' => 'Hello World!' ],
            'goodbye' => [ 'string' => 'Goodbye Cruel World!' ]
        ];
    }

}
use PhpBench\Attributes as Bench;

class AttributedBench
{
    #[Bench\ParamProviders('provideStringsAsArray')]
    public function benchIterable(array $params): void
    {
        $helloThenGoodbye = $params['string'];
    }

    public function provideStringsAsArray(): array
    {
        return [
            'hello' => [ 'string' => 'Hello World!' ],
            'goodbye' => [ 'string' => 'Goodbye Cruel World!' ]
        ];
    }

}

Warning

It should be noted that Generators are consumed completely before the subject is executed. If you have a very large data set, it will be read completely into memory.

Multiple parameter providers can be used, in which case the data sets will be combined into a cartesian product - all possible combinations of the parameters will be generated:

/**
 */
class AnnotatedBench
{
    /**
     * @ParamProviders({"provideStrings", "provideNumbers"})
     */
    public function benchHash($params)
    {
        hash($params['algorithm'], $params['string']);
    }

    public function provideStrings()
    {
        yield 'hello' => [ 'string' => 'Hello World!' ];
        yield 'goodbye' => [ 'string' => 'Goodbye Cruel World!' ];
    }

    public function provideNumbers()
    {
        yield 'md5' => [ 'algorithm' => 'md5' ];
        yield 'sha1' => [ 'algorithm' => 'sha1' ];
    }
 
}
use PhpBench\Attributes as Bench;

class AttributedBench
{
    #[Bench\ParamProviders(['provideStrings', 'provideNumbers'])]
    public function benchHash(array $params)
    {
        hash($params['algorithm'], $params['string']);
    }

    public function provideStrings()
    {
        yield 'hello' => [ 'string' => 'Hello World!' ];
        yield 'goodbye' => [ 'string' => 'Goodbye Cruel World!' ];
    }

    public function provideNumbers()
    {
        yield 'md5' => [ 'algorithm' => 'md5' ];
        yield 'sha1' => [ 'algorithm' => 'sha1' ];
    }
 
}

Will result in the following parameter benchmark scenarios:

// #0
['string' => 'Hello World!', 'algorithm' => 'md5'];

// #1
['string' => 'Goodbye Cruel World!', 'algorithm' => 'md5'[;

// #2
['string' => 'Hello World!', 'algorithm' => 'sha1'];

// #3
['string' => 'Goodbye Cruel World!', 'algorithm' => 'sha1'];

Groups

You can assign benchmark subjects to groups using the @Groups annotation.

/**
 */
class AnnotatedBench
{
    /**
     * @Groups({"one", "two"})
     */
    public function benchTimeItself(): void
    {
        usleep(50);
    }

}
use PhpBench\Attributes as Bench;

class AttributedBench
{
    #[Bench\Groups(["one", "two"])]
    public function benchTimeItself(): void
    {
        usleep(50);
    }

}

The group can then be targeted using the command line interface.

Skipping Subjects

You can skip subjects by using the @Skip annotation:

/**
 */
class AnnotatedBench
{
    /**
     * @Skip
     */
    public function benchThisWillBeSkipped()
    {
    }

}
use PhpBench\Attributes as Bench;

class AttributedBench
{
     #[Bench\Skip]
    public function benchThisWillBeSkipped()
    {
    }

}

Sleeping

Sometimes it may be necessary to pause between iterations in order to let the system recover. Use the @Sleep annotation, specifying the number of microseconds required:

/**
 */
class AnnotatedBench
{
    /**
     * @Sleep(1000)
     */
    public function benchTimeItself(): void
    {
        usleep(50);
    }

}
use PhpBench\Attributes as Bench;

class AttributedBench
{
    #[Bench\Sleep(1000)]
    public function benchTimeItself(): void
    {
        usleep(50);
    }

}

The above example will pause (sleep) for 1 millisecond after each iteration.

Note

This can be overridden using the --sleep option from the CLI.

Time Units

Specify output time units using the @OutputTimeUnit annotation (precision is optional):

/**
 */
class AnnotatedBench
{
    /**
     * @OutputTimeUnit("milliseconds")
     */
    public function benchTimeItself(): void
    {
        usleep(50);
    }

}
use PhpBench\Attributes as Bench;

class AttributedBench
{
    #[Bench\OutputTimeUnit('milliseconds')]
    public function benchTimeItself(): void
    {
        usleep(50);
    }

}

The following time units are available:

  • microseconds
  • milliseconds
  • seconds
  • minutes
  • hours
  • days

Throughput Representation

The output mode determines how the measurements are presented, either time or throughput. time mode is the default and shows the average execution time of a single revolution. throughput shows how many operations are executed within a single time unit:

/**
 */
class AnnotatedBench
{
    /**
     * @OutputTimeUnit("seconds")
     * @OutputMode("throughput")
     */
    public function benchTimeItself(): void
    {
        usleep(50);
    }

}
use PhpBench\Attributes as Bench;

class AttributedBench
{
    #[Bench\OutputTimeUnit('seconds')]
    #[Bench\OutputMode('throughput')]
    public function benchTimeItself(): void
    {
        usleep(50);
    }

}

PHPBench will then render all measurements for benchTimeItself similar to 363,874.536ops/s.

Warm Up

Use the @Warmup annotation to execute any number of revolutions before actually measuring the revolutions time.

/**
 */
class AnnotatedBench
{
    /**
     * @Warmup(2)
     */
    public function benchTimeItself(): void
    {
        usleep(50);
    }

}
use PhpBench\Attributes as Bench;

class AttributedBench
{
    #[Bench\Warmup(2)]
    public function benchTimeItself(): void
    {
        usleep(50);
    }

}

As with revolutions, you may also specify an array.

Timeout

Use the @Timeout annotation to specify the maximum number of seconds before an iteration timesout and fails. The following example will fail after 0.1 seconds:

/**
 */
class AnnotatedBench
{
    /**
     * @Timeout(1.0)
     */
    public function benchTimeItself(): void
    {
        usleep(50);
    }

}
use PhpBench\Attributes as Bench;

class AttributedBench
{
    #[Bench\Timeout(1.0)]
    public function benchTimeItself(): void
    {
        usleep(50);
    }

}

Assertions

You can annotate your benchmarks with assertions which will cause PHPBench to report failures and exit with a non-zero exit code if they fail.

For example, assert that the KDE mode is less than 200 microseconds:

/**
 */
class AnnotatedBench
{
    /**
     * @Assert("mode(variant.time.avg) < 200 ms")
     */
    public function benchTimeItself(): void
    {
        usleep(50);
    }

}
use PhpBench\Attributes as Bench;

class AttributedBench
{
    #[Bench\Assert('mode(variant.time.avg) < 200 ms')]
    public function benchTimeItself(): void
    {
        usleep(50);
    }

}

You can also specify assertions from the command line:

$ phpbench run --assert='mode(variant.time.avg) < 10 hours'

See Assertions for more information.

Format

Override how the variant results are formatted in the progress output.

/**
 */
class AnnotatedBench
{
    /**
     * @Format("mode(variant.time.avg) as ms ~ 'Hello World'")
     */
    public function benchTimeItself(): void
    {
        usleep(50);
    }

}
use PhpBench\Attributes as Bench;

class AttributedBench
{
    #[Bench\Format('mode(variant.time.avg) ~ " Hello World"')]
    public function benchTimeItself(): void
    {
        usleep(50);
    }

}

You can also specify assertions from the command line:

$ phpbench run --format='"This is my time: " ~ mode(variant.time.avg)'

See Expression Language for details on using the expressio language.

RetryThreshold

Set the retry threshold (the deviation beyond which a sample will be considered invalid and retried).

Use to create more stable sets of iterations.

/**
 */
class AnnotatedBench
{
    /**
     * @RetryThreshold(20.0)
     */
    public function benchTimeItself(): void
    {
        usleep(50);
    }

}
use PhpBench\Attributes as Bench;

class AttributedBench
{
    #[Bench\RetryThreshold(20.0)]
    public function benchTimeItself(): void
    {
        usleep(50);
    }

}

Good values are generally 10 or less, the above threshold is 20 because the examples are executed in the continuous integration environment and may cause delays.