Annotributes¶
Configure your benchmarks with Annotations or, if you have PHP 8, Attributes. Note that some of these settings (e.g. runner.revs, runner.iterations) can be set globaly in the Configuration.
Revolutions¶
When testing units of code where microsecond accuracy is important, it is necessary to increase the number of revolutions performed by the benchmark runner. The term “revolutions” (invented here) refers to the number of times the benchmark is executed consecutively within a single time measurement.
We can arrive at a more accurate measurement by determining the mean time
from multiple revolutions (i.e. time / revolutions
) than we could with a
single revolution. In other words, more revolutions means more precision.
Revolutions can be specified using the @Revs
annotation:
/**
*/
class AnnotatedBench
{
/**
* @Revs(10)
*/
public function benchTimeItself(): void
{
usleep(50);
}
}
use PhpBench\Attributes as Bench;
class AttriutedBench
{
#[Bench\Revs(10)]
public function benchTimeItself(): void
{
usleep(50);
}
}
You may also specify an array:
/**
* @Revs({1, 8, 64, 4096})
*/
class HashBench
{
// ...
}
Revolutions can also be overridden from the command line.
Iterations¶
Iterations specify how many samples should be taken - i.e. how many times we run the revolutions and capture time and memory information (for example).
By looking at the separate time measurement of each iteration we can determine how stable the readings are. The less the measurements differ from each other, the more stable the benchmark.
Note
In a perfect environment the readings would all be exactly the same - but such an environment is unlikely to exist
Iterations can be specified using the @Iterations
annotation:
/**
*/
class AnnotatedBench
{
/**
* @Iterations(10)
*/
public function benchTimeItself(): void
{
usleep(50);
}
}
use PhpBench\Attributes as Bench;
class AttriutedBench
{
#[Bench\Iterations(10)]
public function benchTimeItself(): void
{
usleep(50);
}
}
As with revolutions, you may also specify an array.
Iterations can also be overridden from the command line.
You can instruct PHPBench to continuously run the iterations until the
deviation of each iteration fits within a given margin of error by using the
--retry-threshold
. See RetryThreshold for more information.
Benchmark Hooks¶
Method hooks¶
Any number of methods can be executed both before and after each benchmark
subject using the @BeforeMethods
and
@AfterMethods
annotations. Before methods are useful for bootstrapping
your environment:
/**
*/
class AnnotatedBench
{
/**
* @BeforeMethods("setUp")
* @AfterMethods("tearDown")
*/
public function benchTimeItself(): void
{
usleep(50);
}
public function setUp(): void
{
// do somrthing before the benchmark
}
public function tearDown(): void
{
// do somrthing after the benchmark
}
}
use PhpBench\Attributes as Bench;
class AttriutedBench
{
#[Bench\BeforeMethods('setUp')]
#[Bench\AfterMethods("tearDown")]
public function benchTimeItself(): void
{
usleep(50);
}
public function setUp(): void
{
// do somrthing before the benchmark
}
public function tearDown(): void
{
// do somrthing after the benchmark
}
}
Multiple before and after methods can be specified.
Note
If before and after methods are used when the @ParamProviders
annotations are used, then they will also be passed the parameters.
Class Hooks¶
Sometimes you will want to perform actions which establish an external state. For example, creating or populating a database, creating files, etc.
This can be achieved by creating static methods within your benchmark
class and adding the @BeforeClassMethods
and @AfterClassMethods
:
These methods will be executed by the runner once per benchmark class.
/**
* @BeforeClassMethods("setUpBeforeClass")
* @AfterClassMethods("tearDownAfterClass")
*/
class AnnotatedBench
{
/**
*/
public function benchTimeItself(): void
{
usleep(50);
}
public static function setUpBeforeClass(): void
{
// do somrthing before the benchmark
}
public static function tearDownAfterClass(): void
{
// do somrthing after the benchmark
}
}
use PhpBench\Attributes as Bench;
#[Bench\BeforeClassMethods(['setUpBeforeClass'])]
#[Bench\AfterClassMethodsClassMethods(['tearDownBeforeClass'])]
class AttriutedBench
{
public function benchTimeItself(): void
{
usleep(50);
}
public static function setUpBeforeClass(): void
{
// do somrthing before the benchmark
}
public static function tearDownAfterClass(): void
{
// do somrthing after the benchmark
}
}
Note
These methods are static and are executed in a process that is separate from that from which your iterations will be executed. Therefore state will not be carried over to your iterations!.
Parameterized Benchmarks¶
Parameter sets can be provided to benchmark subjects:
/**
*/
class AnnotatedBench
{
/**
* @ParamProviders("provideMd5")
*/
public function benchMd5(array $params): void
{
hash('md5', $params['string']);
}
public function provideMd5(): Generator
{
yield 'hello' => [ 'string' => 'Hello World!' ];
yield 'goodbye' => [ 'string' => 'Goodbye Cruel World!' ];
}
}
use PhpBench\Attributes as Bench;
class AttriutedBench
{
#[Bench\ParamProviders(['provideMd5'])]
public function benchMd5(array $params): void
{
hash('md5', $params['string']);
}
public function provideMd5(): Generator
{
yield 'hello' => [ 'string' => 'Hello World!' ];
yield 'goodbye' => [ 'string' => 'Goodbye Cruel World!' ];
}
}
The benchMd5 subject will now be benchmarked with each parameter set.
The param provider can return a set of parameters using any iterable. For example the above could also be returned as an array:
/**
*/
class AnnotatedBench
{
/**
* @ParamProviders({"provideStringsAsArray"})
*/
public function benchIterable(array $params): void
{
$helloThenGoodbye = $params['string'];
}
public function provideStringsAsArray(): array
{
return [
'hello' => [ 'string' => 'Hello World!' ],
'goodbye' => [ 'string' => 'Goodbye Cruel World!' ]
];
}
}
use PhpBench\Attributes as Bench;
class AttriutedBench
{
#[Bench\ParamProviders('provideStringsAsArray')]
public function benchIterable(array $params): void
{
$helloThenGoodbye = $params['string'];
}
public function provideStringsAsArray(): array
{
return [
'hello' => [ 'string' => 'Hello World!' ],
'goodbye' => [ 'string' => 'Goodbye Cruel World!' ]
];
}
}
Warning
It should be noted that Generators are consumed completely before the subject is executed. If you have a very large data set, it will be read completely into memory.
Multiple parameter providers can be used, in which case the data sets will be combined into a cartesian product - all possible combinations of the parameters will be generated:
/**
*/
class AnnotatedBench
{
/**
* @ParamProviders({"provideStrings", "provideNumbers"})
*/
public function benchHash($params)
{
hash($params['algorithm'], $params['string']);
}
public function provideStrings()
{
yield 'hello' => [ 'string' => 'Hello World!' ];
yield 'goodbye' => [ 'string' => 'Goodbye Cruel World!' ];
}
public function provideNumbers()
{
yield 'md5' => [ 'algorithm' => 'md5' ];
yield 'sha1' => [ 'algorithm' => 'sha1' ];
}
}
use PhpBench\Attributes as Bench;
class AttriutedBench
{
#[Bench\ParamProviders(['provideStrings', 'provideNumbers'])]
public function benchHash(array $params)
{
hash($params['algorithm'], $params['string']);
}
public function provideStrings()
{
yield 'hello' => [ 'string' => 'Hello World!' ];
yield 'goodbye' => [ 'string' => 'Goodbye Cruel World!' ];
}
public function provideNumbers()
{
yield 'md5' => [ 'algorithm' => 'md5' ];
yield 'sha1' => [ 'algorithm' => 'sha1' ];
}
}
Will result in the following parameter benchmark scenarios:
// #0
['string' => 'Hello World!', 'algorithm' => 'md5'];
// #1
['string' => 'Goodbye Cruel World!', 'algorithm' => 'md5'[;
// #2
['string' => 'Hello World!', 'algorithm' => 'sha1'];
// #3
['string' => 'Goodbye Cruel World!', 'algorithm' => 'sha1'];
Groups¶
You can assign benchmark subjects to groups using the @Groups
annotation.
/**
*/
class AnnotatedBench
{
/**
* @Groups({"one", "two"})
*/
public function benchTimeItself(): void
{
usleep(50);
}
}
use PhpBench\Attributes as Bench;
class AttriutedBench
{
#[Bench\Groups(["one", "two"])]
public function benchTimeItself(): void
{
usleep(50);
}
}
The group can then be targeted using the command line interface.
Skipping Subjects¶
You can skip subjects by using the @Skip
annotation:
/**
*/
class AnnotatedBench
{
/**
* @Skip
*/
public function benchThisWillBeSkipped()
{
}
}
use PhpBench\Attributes as Bench;
class AttriutedBench
{
#[Bench\Skip]
public function benchThisWillBeSkipped()
{
}
}
Sleeping¶
Sometimes it may be necessary to pause between iterations in order to let
the system recover. Use the @Sleep
annotation, specifying the number of
microseconds required:
/**
*/
class AnnotatedBench
{
/**
* @Sleep(1000)
*/
public function benchTimeItself(): void
{
usleep(50);
}
}
use PhpBench\Attributes as Bench;
class AttriutedBench
{
#[Bench\Sleep(1000)]
public function benchTimeItself(): void
{
usleep(50);
}
}
The above example will pause (sleep) for 1 millisecond after each iteration.
Note
This can be overridden using the --sleep
option from the CLI.
Time Units¶
Specify output time units using the @OutputTimeUnit
annotation
(precision is optional):
/**
*/
class AnnotatedBench
{
/**
* @OutputTimeUnit("milliseconds")
*/
public function benchTimeItself(): void
{
usleep(50);
}
}
use PhpBench\Attributes as Bench;
class AttriutedBench
{
#[Bench\OutputTimeUnit('milliseconds')]
public function benchTimeItself(): void
{
usleep(50);
}
}
The following time units are available:
microseconds
milliseconds
seconds
minutes
hours
days
Throughput Representation¶
The output mode determines how the measurements are presented, either time or throughput. time mode is the default and shows the average execution time of a single revolution. throughput shows how many operations are executed within a single time unit:
/**
*/
class AnnotatedBench
{
/**
* @OutputTimeUnit("seconds")
* @OutputMode("throughput")
*/
public function benchTimeItself(): void
{
usleep(50);
}
}
use PhpBench\Attributes as Bench;
class AttriutedBench
{
#[Bench\OutputTimeUnit('seconds')]
#[Bench\OutputMode('throughput')]
public function benchTimeItself(): void
{
usleep(50);
}
}
PHPBench will then render all measurements for benchTimeItself similar to 363,874.536ops/s.
Warm Up¶
Use the @Warmup
annotation to execute any number of revolutions before
actually measuring the revolutions time.
/**
*/
class AnnotatedBench
{
/**
* @Warmup(2)
*/
public function benchTimeItself(): void
{
usleep(50);
}
}
use PhpBench\Attributes as Bench;
class AttriutedBench
{
#[Bench\Warmup(2)]
public function benchTimeItself(): void
{
usleep(50);
}
}
As with revolutions, you may also specify an array.
Timeout¶
Use the @Timeout
annotation to specify the maximum number of seconds
before an iteration timesout and fails. The following example will fail after
0.1 seconds:
/**
*/
class AnnotatedBench
{
/**
* @Timeout(1.0)
*/
public function benchTimeItself(): void
{
usleep(50);
}
}
use PhpBench\Attributes as Bench;
class AttriutedBench
{
#[Bench\Timeout(1.0)]
public function benchTimeItself(): void
{
usleep(50);
}
}
Assertions¶
You can annotate your benchmarks with assertions which will cause PHPBench to report failures and exit with a non-zero exit code if they fail.
For example, assert that the KDE mode is less than 200 microseconds:
/**
*/
class AnnotatedBench
{
/**
* @Assert("mode(variant.time.avg) < 200 ms")
*/
public function benchTimeItself(): void
{
usleep(50);
}
}
use PhpBench\Attributes as Bench;
class AttriutedBench
{
#[Bench\Assert('mode(variant.time.avg) < 200 ms')]
public function benchTimeItself(): void
{
usleep(50);
}
}
You can also specify assertions from the command line:
$ phpbench run --assert='mode(variant.time.avg) < 10 hours'
See Assertions for more information.
Format¶
Override how the variant results are formatted in the progress output.
/**
*/
class AnnotatedBench
{
/**
* @Format("mode(variant.time.avg) as ms ~ \" Hello World\"")
*/
public function benchTimeItself(): void
{
usleep(50);
}
}
use PhpBench\Attributes as Bench;
class AttriutedBench
{
#[Bench\Format('mode(variant.time.avg) ~ " Hello World"')]
public function benchTimeItself(): void
{
usleep(50);
}
}
You can also specify assertions from the command line:
$ phpbench run --format='"This is my time: " ~ mode(variant.time.avg)'
See Expression Language for details on using the expressio language.
RetryThreshold¶
Set the retry threshold (the deviation beyond which a sample will be considered invalid and retried).
Use to create more stable sets of iterations.
/**
*/
class AnnotatedBench
{
/**
* @RetryThreshold(20.0)
*/
public function benchTimeItself(): void
{
usleep(50);
}
}
use PhpBench\Attributes as Bench;
class AttriutedBench
{
#[Bench\RetryThreshold(20.0)]
public function benchTimeItself(): void
{
usleep(50);
}
}
Good values are generally 10 or less, the above threshold is 20 because the examples are executed in the continuous integration environment and may cause delays.