Continuous Integration

Automatically detecting performance regressions is a holy grail for benchmarking enthusiasts. Unfortunately your CI servers are almost certainly unstable environments which makes them unsuitable for picking up subtle changes in performance. You can, however, reliably detect major performance regressions and, if monitoring performance over time, identfy performance trends.

The following methods are unofficial recommendations and, where applicable, you use them at your own risk.

Github Actions Benchmarks

The github-action-benchmark is a github action that uses github pages to store historical benchmarking results and builds a static page containing all the results.

In order to use it you need to configure a report in PHPBench, introduce the followng report confguration` in phpbench.json:

{
    "report.generators": {
        "github-action-benchmark": {
            "title": "github-action-benchmark",
            "description": "https://github.com/benchmark-action/github-action-benchmark",
            "cols": [
                "name",
                "unit",
                "value",
                "range",
                "extra"
            ],
            "expressions": {
                "name": "format('%s::%s%s', first(benchmark_name),first(subject_name),if(first(variant_name) != false, format(' (%s)', first(variant_name)), ''))",
                "unit": "time_unit(first(subject_time_unit), true)",
                "value": "time_convert(mode(result_time_avg), null, time_unit(first(subject_time_unit)))",
                "range": "format('± %.2f%%', rstdev(result_time_avg))",
                "extra": "format('%d iterations, %d revs', first(variant_iterations), first(variant_revs))"
            },
            "aggregate": [
                "suite_tag",
                "benchmark_class",
                "subject_name",
                "variant_index"
            ],
            "generator": "expression"
        }
    }
}

You can now generate the JSON document required by the github action:

$ phpbench run --report=github-action-benchmark --output=json > output.json

Now introduce the github workflow:

name: Benchmark

on:
  push:
    branches:
      # change this to your main branch
      - master

jobs:
  benchmark:
    name: Benchmark
    runs-on: ubuntu-latest
    permissions:
      contents: write
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Install PHP
        uses: shivammathur/setup-php@v2
        with:
          coverage: none
          php-version: "8.2"
          tools: composer:v2

      - name: Composer install
        uses: ramsey/composer-install@v2
        with:
          composer-options: "--no-scripts"

      # optionially ensure that the gh-pages branch exists. feel free to create
      # this yourself and ommit this step.
      - name: Ensure gh-pages branch exists
        run: |
          if ! git ls-remote --exit-code --heads origin gh-pages > /dev/null 2>&1; then
            git config user.name "github-actions[bot]"
            git config user.email "12345678+github-actions[bot]@users.noreply.github.com"
            empty_tree="$(git hash-object -t tree /dev/null)"
            commit="$(git commit-tree "$empty_tree" -m 'Initial gh-pages branch')"
            git push origin "$commit:refs/heads/gh-pages"
          fi

      - name: Run PHPBench
        run: bin/phpbench run --progress=plain --report=github-action-benchmark --output=json > output.json

      - name: Store benchmark result
        uses: benchmark-action/github-action-benchmark@v1
        with:
          name: PHPBench Performance
          tool: customSmallerIsBetter
          output-file-path: output.json
          gh-pages-branch: gh-pages
          benchmark-data-dir-path: benchmarks
          auto-push: true
          github-token: ${{ secrets.GITHUB_TOKEN }}

In the above example we publish the results to the benchmarks. The PHPBench report can be seen here: https://phpbench.github.io/phpbench/benchmarks/

Note

The above workflow only runs on the main branch and does not provide any feedback on pull requests. The github actin can also automatically comment on pull requests when regressions are detected. See the action’s README file for more information.