A few years in the past, Laravel launched a terrific function which permits to run PHPUnit / Pest checks in parallel. This ends in an enormous increase in efficiency.
By default, it determines the concurrency stage by looking on the variety of CPU cores your machine has. So, in case you’re utilizing a contemporary Mac that has 10 CPU cores, it should run 10 checks on the similar time, vastly reducing down on the time your testsuite must run utterly.
A default runner on GitHub would not have that many cores, so you may’t leverage parallel testing pretty much as good as in your typical native environments.
On this weblog publish, I might like to point out you a manner of operating your checks on GitHub, by splitting them up in small chunks that may run concurrently.
We use this method at Oh Expensive to chop down the operating time of our huge testsuite from 16 minutes to solely simply 4. On this weblog publish all examples will come from the Oh Expensive code base.
What we will do
Like already talked about within the introduction, a typical take a look at runner on GitHub hasn’t received loads of cores, that means the default manner of operating checks in parallel that Laravel gives (operating a take a look at per CPU core) would not work effectively.
What you are able to do at GitHub nonetheless is operating loads of GitHub actions jobs in parallel. So what we’re going to do is splitting our testsuite in equal chunks, and create a take a look at job on GitHub motion per chunk. These chunks can run in parallel, which is able to immensely lower the full operating time.
I am going to inform you how you can obtain this technically within the the rest of this weblog publish, however this is already how the top consequence will appear like on GitHub.
Within the screenshot above you may see that our take a look at suite is cut up up in 12 elements which is able to all run concurrently. Composer / NPM will solely run as soon as to construct up the dependencies and property and they are going to be utilized in all 12 testing elements.
Splitting up the testsuite in equal elements
Let’s first check out how we are able to cut up the testsuite in equal elements. Oh Expensive makes use of Pest as a take a look at runner, which gives a --list-tests choice to output all checks.
This is a little bit of code to get all take a look at class names from that output.
$course of = new Course of([__DIR__ . '/../vendor/bin/pest', '--list-tests']); $course of->mustRun(); $index = $shardNumber - 1; $allTestNames = Str::of($course of->getOutput()) ->explode("n") ->filter(fn(string $line) => str_contains($line, ' - ')) ->map(operate (string $fullTestName) { $testClassName = Str::of($fullTestName) ->exchange('- ', '') ->trim() ->between('\', '::') ->afterLast('') ->toString(); return $testClassName; }) ->filter() ->distinctive();
In $allTestNames might be a set containing all class names (= file names) which might be contained in the take a look at suite.
To separate the gathering up in a number of elements, you should use the cut up operate which accepts the variety of elements you need. This is how you’d cut up up the checks in 12 elements, and get the primary half
$testNamesOfFirstPart = $allTestNames ->cut up(12) ->get(key: 0)
PHPUnit / Pest additionally gives a --filter choice to solely run particular checks. For those who solely wish to run the checks in from the ArchTest class (which is displayed within the screenshot above), you can execute this.
# Will solely run the checks from the ArchTest file vendor/bin/pest --filter=ArchTest
You should utilize | to specify a number of patterns. This is how you can execute the checks from a number of information
# Will solely run the checks from the ArchTest file vendor/bin/pest --filter=ArchTest|CheckSitesBeingMonitoredTest
This is how you can use the $testNamesOfFirstPart from the earlier snippet to run the primary a part of the checks programmatically.
$course of = new Course of( command: [ './vendor/bin/pest', '--filter', $testNamesOfFirstPart->join('|')], timeout: null ); $course of->begin(); foreach ($course of as $knowledge) { echo $knowledge; } $course of->wait(); exit($course of->getExitCode());
Operating the testsuite elements in parallel at GitHub
Now that you know the way you can cut up up a take a look at suite in equal elements, let’s check out how we are able to run all these elements in parallel on GitHub actions.
GitHub actions assist [a matrix parameter](TODO: add hyperlink). Shortly stated, this matrix parameter is used for testing variations of your take a look at suite, and it’ll run these variations concurrently.
This is the a part of the Oh Expensive GitHub workflow the place the matrix is being arrange. I’ve omitted a number of elements for brevity.
identify: Run checks jobs: run-tests: identify: Run Exams (Half ${{ matrix.shard_number }}/${{ matrix.total_shard_count }}) runs-on: ubuntu-latest technique: matrix: total_shard_count: [12] shard_number: ['01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12'] steps: - identify: Run checks run: ./vendor/bin/pest
The matrix will create jobs per mixture within the matrix. So it should run 1 (just one ingredient in total_shard_count) * 12 (twelve parts in shard_number) = 12 occasions.
Within the Run checks step, pest is executed. It will end in the entire testsuite being executed 12 occasions. In fact we do not wish to execute the entire take a look at suite 12 occasions, however solely every separate 1/12 a part of the testsuite.
We will obtain this by not operating /vendor/bin/pest however a customized PHP script known as github_parallel_test_runner that may obtain the total_shard_count and the shard_number as setting variables.
identify: Run checks jobs: run-tests: identify: Run Exams (Half ${{ matrix.shard_number }}/${{ matrix.total_shard_count }}) runs-on: ubuntu-latest technique: matrix: total_shard_count: [12] shard_number: ['01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12'] steps: - identify: Run checks run: ./bin/github_parallel_test_runner env: TOTAL_SHARD_COUNT: ${{ matrix.total_shard_count }} SHARD_NUMBER: ${{ matrix.shard_number }}
This is the content material of ./bin/github_parallel_test_runner in our code base. It is going to learn the setting variables, execute Pest utilizing the --list-files and --filter flags to solely run part of the checks like defined within the earlier part of this publish.
#!/usr/bin/env php <?php use IlluminateSupportCollection; use IlluminateSupportStr; use SymfonyComponentProcessProcess; require_once 'vendor/autoload.php'; $shardNumber = (int)getenv('SHARD_NUMBER'); $totalShardCount = (int)getenv('TOTAL_SHARD_COUNT'); if ($shardNumber === 0 || $totalShardCount === 0) { echo "SHARD_NUMBER and TOTAL_SHARD_COUNT should be set." . PHP_EOL; exit(1); } new ParallelTests($totalShardCount)->run($shardNumber); class ParallelTests { public operate __construct( protected int $totalShardCount, ) { } public operate run(int $shardNumber): by no means { $testNames = $this->getTestNames($shardNumber); echo "Operating {$testNames->rely()} checks on node {$shardNumber} of {$this->totalShardCount}..." . PHP_EOL; $exitCode = $this->runPestTests($testNames); exit($exitCode); } protected operate getTestNames(int $shardNumber): Assortment { $course of = new Course of([__DIR__ . '/../vendor/bin/pest', '--list-tests']); $course of->mustRun(); $index = $shardNumber - 1; $allTestNames = Str::of($course of->getOutput()) ->explode("n") ->filter(fn(string $line) => str_contains($line, ' - ')) ->map(operate (string $fullTestName) { $testClassName = Str::of($fullTestName) ->exchange('- ', '') ->trim() ->between('\', '::') ->afterLast('') ->toString(); return $testClassName; }) ->filter() ->distinctive(); echo "Detected {$allTestNames->rely()} checks:" . PHP_EOL; return $allTestNames ->cut up($this->totalShardCount) ->get($index); } protected operate runPestTests(Assortment $testNames): ?int { $course of = new Course of( command: ['./vendor/bin/pest', '--filter', $testNames->join('|')], timeout: null ); $course of->begin(); foreach ($course of as $knowledge) { echo $knowledge; } $course of->wait(); return $course of->getExitCode(); } }
To make this script executable you could execute this command and push the modifications permissions…
chmod +x ./bin/github_parallel_test_runner
Solely run composer and NPM / Yarn as soon as
Within the screenshot above, you can see how there is a “Composer and Yarn” step that’s executing solely as soon as earlier than all of the take a look at elements run. This is that screenshot once more.
A GitHub motion workflow can include a number of jobs, and you’ll outline dependencies between them. Within the snippet beneath you will see the setup-dependencies job being outlined (I’ve omitted all steps relating to to Yarn / NPM to maintain issues temporary). We save the vendor listing as an artifact and use that saved listing in all of our take a look at jobs. Lastly, there’s additionally a step to wash up any created artifacts.
You may see that within the wants key, you may outline the steps {that a} job is dependent upon.
jobs: setup-dependencies: identify: Composer and Yarn runs-on: ubuntu-latest steps: - makes use of: actions/checkout@v3 with: fetch-depth: 1 - identify: Setup PHP makes use of: shivammathur/setup-php@v2 with: php-version: 8.4 extensions: dom, curl, libxml, mbstring, zip, pcntl, pdo, sqlite, pdo_sqlite, bcmath, cleaning soap, intl, gd, exif, iconv, imagick protection: none - identify: Set up composer dependencies run: composer set up --prefer-dist --no-scripts -q -o - identify: Add vendor listing makes use of: actions/upload-artifact@v4 with: identify: vendor-directory-${{ github.run_id }} path: vendor retention-days: 1 run-tests: wants: [ setup-dependencies] identify: Run Exams (Half ${{ matrix.shard_number }}/${{ matrix.total_shard_count }}) runs-on: ubuntu-latest technique: fail-fast: false matrix: total_shard_count: [12] shard_number: ['01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12'] steps: - makes use of: actions/checkout@v3 with: fetch-depth: 1 - identify: Obtain vendor listing makes use of: actions/download-artifact@v4 with: identify: vendor-directory-${{ github.run_id }} path: vendor - identify: Run checks run: ./bin/github-parallel-test-runner env: TOTAL_SHARD_COUNT: ${{ matrix.total_shard_count }} SHARD_NUMBER: ${{ matrix.shard_number }} cleanup-artifacts: identify: Clear up artifacts wants: [setup-dependencies, run-tests] runs-on: ubuntu-latest if: at all times() steps: - identify: Delete artifacts makes use of: geekyeggo/delete-artifact@v2 with: identify: | vendor-directory-${{ github.run_id }} failOnError: false
Cancelling stale checks
There’s one other neat little factor that we do in our GitHub motion workflow to avoid wasting time. At any time when modifications are pushed to a sure department, we’re not likely within the outcomes of the checks of any earlier commits / pushes on that department anymore.
Would not it’s good if the take a look at for any older commits on a department had been robotically cancelled, so the checks for the brand new commits / push would instantly begin?
Effectively, with this snippet in your workflow, that is precisely what’s going to occur.
identify: Run checks on: push: paths: - '**.php' - '.github/workflows/run-tests.yml' - 'phpunit.xml.dist' - 'composer.json' - 'composer.lock' concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true jobs:
Our full take a look at workflow
To make issues clear, this is our total workflow file together with the setup of all of the companies that our testsuite wants like MySQL, Redis, ClickHouse, Lighthouse and extra.
identify: Run checks on: push: paths: - '**.php' - '.github/workflows/run-tests.yml' - 'phpunit.xml.dist' - 'composer.json' - 'composer.lock' concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true jobs: setup-dependencies: identify: Composer and Yarn runs-on: ubuntu-latest steps: - makes use of: actions/checkout@v3 with: fetch-depth: 1 - identify: Setup PHP makes use of: shivammathur/setup-php@v2 with: php-version: 8.4 extensions: dom, curl, libxml, mbstring, zip, pcntl, pdo, sqlite, pdo_sqlite, bcmath, cleaning soap, intl, gd, exif, iconv, imagick protection: none - identify: Get Composer Cache Listing id: composer-cache run: echo "dir=$(composer config cache-files-dir)" >> $GITHUB_OUTPUT - identify: Cache Composer dependencies makes use of: actions/cache@v3 with: path: ${{ steps.composer-cache.outputs.dir }} key: ${{ runner.os }}-composer-${{ hashFiles('**/composer.lock') }} restore-keys: ${{ runner.os }}-composer- - identify: Set up composer dependencies run: composer set up --prefer-dist --no-scripts -q -o - identify: Add vendor listing makes use of: actions/upload-artifact@v4 with: identify: vendor-directory-${{ github.run_id }} path: vendor retention-days: 1 - identify: Cache Node Modules makes use of: actions/cache@v3 with: path: node_modules key: ${{ runner.os }}-node-${{ hashFiles('**/yarn.lock') }} restore-keys: ${{ runner.os }}-node- - identify: Cache constructed property makes use of: actions/cache@v3 with: path: | public/construct public/scorching public/css public/js key: ${{ runner.os }}-assets-${{ hashFiles('assets/**/*') }}-${{ hashFiles('**/yarn.lock') }} restore-keys: ${{ runner.os }}-assets- - identify: Compile property run: | yarn set up --pure-lockfile yarn construct if: steps.node-cache.outputs.cache-hit != 'true' || steps.assets-cache.outputs.cache-hit != 'true' - identify: Add compiled property makes use of: actions/upload-artifact@v4 with: identify: compiled-assets-${{ github.run_id }} path: | public/construct public/scorching public/css public/js retention-days: 1 install-chrome-and-lighthouse: identify: Set up Chrome and Lighthouse runs-on: ubuntu-latest steps: - makes use of: actions/checkout@v3 with: fetch-depth: 1 - identify: Setup PHP makes use of: shivammathur/setup-php@v2 with: php-version: 8.4 extensions: dom, curl, libxml, mbstring, zip, pcntl, pdo, sqlite, pdo_sqlite, bcmath, cleaning soap, intl, gd, exif, iconv, imagick protection: none - identify: Setup downside matchers run: | echo "::add-matcher::${{ runner.tool_cache }}/php.json" echo "::add-matcher::${{ runner.tool_cache }}/phpunit.json" - identify: Set up Chrome Launcher run: npm set up chrome-launcher - identify: Set up Lighthouse run: npm set up lighthouse - identify: Cache take a look at setting makes use of: actions/upload-artifact@v4 with: identify: test-env-${{ github.run_id }} path: | node_modules retention-days: 1 run-tests: wants: [ setup-dependencies, install-chrome-and-lighthouse ] identify: Run Exams (Half ${{ matrix.shard_number }}/${{ matrix.total_shard_count }}) runs-on: ubuntu-latest technique: fail-fast: false matrix: total_shard_count: [12] shard_number: ['01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12'] companies: mysql: picture: mysql:8.0 env: MYSQL_ALLOW_EMPTY_PASSWORD: sure MYSQL_DATABASE: ohdear_testing ports: - 3306 choices: --health-cmd="mysqladmin ping" --health-interval=10s --health-timeout=5s --health-retries=3 redis: picture: redis ports: - 6379:6379 choices: --entrypoint redis-server clickhouse: picture: clickhouse/clickhouse-server choices: >- --health-cmd "clickhouse shopper -q 'SELECT 1'" --health-interval 10s --health-timeout 5s --health-retries 5 ports: - 8123:8123 - 9000:9000 - 9009:9009 env: CLICKHOUSE_DB: ohdear CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT: 1 steps: - makes use of: actions/checkout@v3 with: fetch-depth: 1 - identify: create db run: | sudo /and so forth/init.d/mysql begin mysql -u root -proot -e 'CREATE DATABASE IF NOT EXISTS ohdear_testing;' - identify: Setup PHP makes use of: shivammathur/setup-php@v2 with: php-version: 8.4 extensions: dom, curl, libxml, mbstring, zip, pcntl, pdo, sqlite, pdo_sqlite, bcmath, cleaning soap, intl, gd, exif, iconv, imagick protection: none - identify: Obtain take a look at setting makes use of: actions/download-artifact@v4 with: identify: test-env-${{ github.run_id }} path: . - identify: Obtain vendor listing makes use of: actions/download-artifact@v4 with: identify: vendor-directory-${{ github.run_id }} path: vendor - identify: Obtain compiled property makes use of: actions/download-artifact@v4 with: identify: compiled-assets-${{ github.run_id }} path: public - identify: Put together Laravel Utility run: | cp .env.instance .env php artisan key:generate - identify: Set permissions for vendor binaries run: chmod -R +x vendor/bin/ - identify: Run checks run: ./bin/github-parallel-test-runner env: DB_PORT: ${{ job.companies.mysql.ports[3306] }} REDIS_PORT: ${{ job.companies.redis.ports[6379] }} CLICKHOUSE_HOST: localhost CLICKHOUSE_PORT: 8123 CLICKHOUSE_DATABASE: ohdear CLICKHOUSE_USERNAME: default CLICKHOUSE_PASSWORD: CLICKHOUSE_HTTPS: false TOTAL_SHARD_COUNT: ${{ matrix.total_shard_count }} SHARD_NUMBER: ${{ matrix.shard_number }} cleanup-artifacts: identify: Clear up artifacts wants: [setup-dependencies, install-chrome-and-lighthouse, run-tests] runs-on: ubuntu-latest if: at all times() steps: - identify: Delete artifacts makes use of: geekyeggo/delete-artifact@v2 with: identify: | vendor-directory-${{ github.run_id }} compiled-assets-${{ github.run_id }} test-env-${{ github.run_id }} failOnError: false
Execs and cons
There are execs and cons for operating checks in parallel on GitHub actions. Let’s begin with crucial professional first: your take a look at suite will run considerably sooner. It is also very straightforward to extend or lower the extent of parallelism. A sooner testsuite signifies that your total suggestions cycle is quicker, which is an enormous win.
On the cons aspect, there is definitely some extra complexity concerned: you want a script to separate checks, the workflow turns into extra advanced.
There’s additionally the danger of uneven take a look at distribution. In case your checks range considerably in execution time, you may find yourself with some shards ending a lot sooner than others, decreasing the effectivity achieve.
In closing
The strategy outlined on this publish works exceptionally effectively for big take a look at suites like now we have at Oh Expensive, the place we have seen important discount in take a look at execution time. However even smaller initiatives can profit from this method, particularly as they develop over time.
This system can also be used at a few massive initiatives we have developed at Spatie, the place my colleague Rias got here up with the thought to separate the take a look at suite.
