Visible UI testing (often known as regression testing) is a testing method to validate that your modifications don’t have any sudden influence on the UI.
Sometimes, such assessments take a picture snapshot of your entire software below check or a selected aspect after which examine the picture to a beforehand authorised baseline picture. If the pictures are the identical (inside a set pixel tolerance), it’s decided that the online software seems the identical to the consumer. If there are variations, then there was some change to the DOM format, fonts, colours, or different visible properties that have to be investigated.
This publish will discover how you can automate visible regression assessments of a “fashionable” internet software utilizing Playwright and GitHub actions. The aim is to construct a testing setup that checks for UI regressions on every pull request and permits selectively updating the baseline pictures when wanted.
Primary data of JavaScript (or TypeScript) and GitHub Actions is beneficial.
We assume you’ll combine this setup into an current internet software. If you wish to strive it from scratch, I like to recommend scaffolding a brand new internet software utilizing Vite.
You will discover a whole instance of the ensuing software in this GitHub repo.
For reference, right here’s what the tiny internet software we’re testing seems like:
Playwright setup
For our testing setup, we’ll use Playwright, an E2E testing framework. I like Playwright as a result of it offers an ideal developer expertise and good defaults out of the field, however you may obtain the identical consequence with comparable instruments corresponding to Cypress or Selenium.
To set up Playwright cd into our venture and run the next command:
npm init playwright
We’ll be prompted with a number of totally different choices (e.g. JavaScript vs TypeScript, and so forth.). When requested, we should always add a GitHub Actions workflow to run our assessments on CI simply:
✔ Add a GitHub Actions workflow? (y/N) · trueAs soon as achieved, Playwright will add an instance check in ./assessments/instance.spec.ts, an instance E2E file in ./tests-examples/demo-todo-app.spec.ts (that we will ignore), and the Playwright configuration file in ./playwright.config.ts.
✔ Success! Created a Playwright Take a look at venture at ~/your-project-dir
Inside that listing, you may run a number of instructions:
npx playwright check
Runs the end-to-end assessments.
npx playwright check --project=chromium
Runs the assessments solely on Desktop Chrome.
npx playwright check instance
Runs the assessments in a selected file.
npx playwright check --debug
Runs the assessments in debug mode.
npx playwright codegen
Auto generate assessments with Codegen.
We advise that you simply start by typing:
npx playwright check
And take a look at the next recordsdata:
- ./assessments/instance.spec.ts - Instance end-to-end check
- ./tests-examples/demo-todo-app.spec.ts - Demo Todo App end-to-end assessments
- ./playwright.config.ts - Playwright Take a look at configuration
Go to https://playwright.dev/docs/intro for extra info. ✨The Playwright configuration file generated by the scaffolding course of (./playwright.config.ts) offers some good defaults, however I like to recommend making use of a few modifications.
First, to simplify our setup, replace the initiatives listing to run our assessments solely on Chromium:
initiatives: [
{
name: "chromium",
use: {
...devices["Desktop Chrome"],
},
},We are able to replace the venture listing in a while, to incorporate extra browsers/units, if wanted.
Then, configure the webServer part in order that Playwright can robotically serve our software once we run our assessments:
webServer: {
command: 'npm run dev --port 8080',
port: 8080,
reuseExistingServer: true
},Generate the preliminary snapshots
The instance check generated by Playwright (./assessments/instance.spec.ts) will not be a visible regression check, so let’s delete its content material and add a tiny check that fits our wants:
import { check, anticipate } from "@playwright/check";
check("instance check", async ({ web page }) => {
await web page.goto("https://mmazzarolo.com/");
await anticipate(web page).toHaveScreenshot();
});For visible regression testing, Playwright consists of the power to provide and visually examine snapshots utilizing await anticipate(web page).toHaveScreenshot(). On first execution, Playwright check will generate reference snapshots. Subsequent runs will examine in opposition to the reference.
We’re lastly able to run your check:
The check runner will say one thing alongside the road of:
Error: instance.spec.ts-snapshots/example-test-1-chromium-darwin.png is lacking in snapshots, writing precise.That’s as a result of there was no baseline picture but, so this methodology took a bunch of snapshots till two consecutive snapshots matched, and saved the final snapshot to file system. It’s now able to be added to the repository (in assessments/instance.spec.ts-snapshots/example-test-1-chromium-darwin.png):
Now, if we run our check once more:
The check ought to now succeed as a result of the present UI matches the UI of the reference snapshot generated within the earlier run.
Replace the snapshots domestically
Let’s transfer to the fascinating half.
If we make any visible change to the UI and rerun our assessments, they may fail, and Playwright will present us a pleasant diff between the “precise” and “anticipated” snapshot:
In instances corresponding to this one, once we need to make some voluntary modifications to a web page, we have to replace the reference snapshots. We are able to do that with the --update-snapshots flag:
npx playwright check --update-snapshots[chromium] › instance.spec.ts:3:1 › instance check
assessments/instance.spec.ts-snapshots/example-test-1-chromium-darwin.png is re-generated, writing precise.Now that we’ve seen how you can run visible assessments and replace the snapshots domestically, we’re prepared to maneuver to the CI stream.
Operating the assessments in CI utilizing GitHub Actions
GitHub Actions is a steady integration and steady supply (CI/CD) platform that permits you to automate your construct, check, and deployment pipeline. With GitHub Actions we will create workflows that construct and check each pull request to a repository.
A great place to begin for a visible regression testing setup is to run the assessments on every pull request creation and replace. Fortunately, Playwright already generated a useful GitHub Actions workflow to run assessments for this particular use case in .github/workflows/playwright.yml.
This workflow ought to work completely high-quality out-of-the-box, however I like to recommend tweaking it a bit to:
- Set up solely the browsers we’re focusing on with our assessments (
--with deps chromium); - Save the check consequence artifacts solely when assessments fail to keep away from storing pointless artifacts (
if: failure()).
title: Playwright Assessments
on:
push:
branches: [main, master]
pull_request:
branches: [main, master]
jobs:
check:
timeout-minutes: 60
runs-on: ubuntu-newest
steps:
- makes use of: actions/checkout@v2
- makes use of: actions/setup-node@v2
with:
node-version: "14.x"
- title: Set up dependencies
run: npm set up
- title: Set up Playwright Browsers
- run: npx playwright set up --with-deps
+ run: npx playwright set up --with-deps chromium
- title: Run Playwright assessments
run: npx playwright check
- makes use of: actions/add-artifact@v2
- if: all the time()
+ if: failure()
with:
title: playwright-report
path: playwright-report/
retention-days: 30As soon as we commit this workflow within the GitHub repo, submitting a brand new pull request will set off the Playwright assessments.
Nonetheless, our check will doubtless fail at this level as a result of the reference snapshots in our repo have been captured in an surroundings that differs from the one used on CI (until we created them on a machine operating Ubuntu).
For extra info on the check consequence, we will examine both the GitHub Motion output or the check artifacts:
For instance, if we generated the reference snapshots on macOS, we’ll obtain an error stating that the snapshots for Linux haven’t been discovered:
Error: assessments/instance.spec.ts-snapshots/example-test-1-chromium-linux.png is lacking in snapshotsLet’s see how we will replace the reference snapshots for the CI surroundings.
Producing the reference snapshots domestically utilizing --update-snapshots was a bit of cake, however on CI, it’s a distinct story as a result of we should determine the place, how, and when to retailer them.
There are many methods we will deal with this stream, however for the sake of simplicity, let’s begin easy.
One sample that has labored nicely for me is to make use of a GitHub Motion workflow to replace the reference snapshots when a selected remark with a “/update-snapshots” textual content is posted in a pull request.
The thought is that each time we submit a pull request that we anticipate to influence the UI, we will publish the “/update-snapshots” remark in order that CI will generate and commit the up to date snapshots within the pull request department.
title: Replace Snapshots
on:
issue_comment:
varieties: [created]
jobs:
updatesnapshots:
if: ${{ github.occasion.problem.pull_request && github.occasion.remark.physique == '/replace-snapshots'}}
timeout-minutes: 60
runs-on: ubuntu-newest
steps:
- makes use of: actions/checkout@v2
with:
fetch-depth: 0
token: ${{ secrets and techniques.GITHUB_TOKEN }}
- title: Get SHA and department title
id: get-department-and-sha
run: |
sha_and_branch=$(
curl
-H 'authorization: Bearer ${{ secrets and techniques.GITHUB_TOKEN }}'
https://api.github.com/repos/${{ github.repository }}/pulls/${{ github.occasion.problem.quantity }}
| jq -r '.head.sha," ",.head.ref');
echo "::set-output title=sha::$(echo $sha_and_branch | reduce -d " " -f 1)";
echo "::set-output title=department::$(echo $sha_and_branch | reduce -d " " -f 2)"
- title: Fetch Department
run: git fetch
- title: Checkout Department
run: git checkout ${{ steps.get-department-and-sha.outputs.department }}
- makes use of: actions/setup-node@v2
with:
node-version: "14.x"
- title: Set up dependencies
run: npm set up
- title: Set up Playwright browsers
run: npx playwright set up --with-deps chromium
- title: Replace snapshots
run: npx playwright check --replace-snapshots --reporter=listing
- makes use of: stefanzweifel/git-auto-commit-motion@v4
with:
commit_message: "[CI] Replace Snapshots"If we publish this workflow, create a brand new pull request, and add a “/update-snapshots” remark, CI will deal with producing the reference snapshots.
Good!
That’s it — now we have totally automated visible regression testing on our Steady Integration. Each time new modifications are posted in new pull requests, our assessments will guarantee we’re not mistakenly making modifications to the UI. We are able to additionally replace our baseline snapshots by posting a remark within the pull request.
In case your venture helps preview hyperlinks (corresponding to those robotically generated by Netlify, Vercel, and so forth.), carry on studying. In any other case, soar to the Conclusion part under.
Run Playwright assessments in opposition to deploy preview (Netlify, Vercel, and so forth.)
In case your internet software runs on platforms corresponding to Netlify and Vercel, it’s doubtless you’re utilizing their “Deploy Preview” characteristic: a strategy to preview pull request modifications with out impacting the online software in manufacturing.
Deploy previews are enabled by default for GitHub pull requests, and so they work by deploying them to a novel URL totally different from the one your manufacturing website makes use of.
In case your internet software makes use of deploy previews, we will combine them into our visible regression testing workflows to match the snapshots in opposition to them as an alternative of the native internet server.
This method brings two advantages. First, we keep away from spinning up a neighborhood webserver only for operating the assessments. Second, by operating our assessments in opposition to a preview hyperlink, we’ll produce extra dependable snapshots as a result of preview hyperlinks are a 1:1 illustration of what we should always see in manufacturing.
From a high-level view, there are three important modifications we have to make in our codebase to run our Playwright assessments in opposition to deploy previews:
- Permit passing the URL to check as a parameter to make our assessments conscious of the deploy preview hyperlink.
- Replace our GitHub Motion workflows to attend for the deploy preview to be accomplished.
- Replace our GitHub Motion workflows to move to Playwright the deploy preview URL (as an surroundings variable).
Right here’s how we will obtain this in Netlify (when you’re utilizing Vercel or another platform that helps deploy preview, the modifications needs to be nearly similar).
First, replace the use.baseURL worth of playwright.config.ts to obtain the deploy URL as an surroundings variable (WEBSITE_URL):
use: {
baseURL: course of.env.WEBSITE_URL,
}Additionally, let’s disable Playwright’s internet server if a WEBSITE_URL surroundings variable is offered:
webServer: course of.env.WEBSITE_URL
? undefined
: {
command: "npm run dev --port 8080",
port: 8080,
reuseExistingServer: true,
},Then, replace our playwright.yaml workflow to run assessments in opposition to the deploy preview.
To attend for the Netlify deploy preview URL, we will use the mmazzarolo/wait-for-netlify-action GitHub Motion.
mmazzarolo/wait-for-netlify-actionis a fork ofprobablyup/wait-for-netlify-action. By default,probablyup/wait-for-netlify-actionassumes it’s operating inside a workflow triggered by a pull request push. In our case, theupdate-snapshots.ymlworkflow is triggered by a remark, so I forked this GitHub Motion to make sure it runs on any workflow, no matter what triggered it.
The wait-for-netlify-action GitHub Motion requires two issues:
- Setting a
NETLIFY_TOKENGitHub Motion secret with a Netlify Private Entry Token. - Passing a Netlify Web site ID (from Netlify: Settings → Web site Particulars → Normal) to the
site_idparameter within the workflow.
title: Playwright Assessments
on:
push:
branches: [main, master]
pull_request:
branches: [main, master]
workflow_run:
workflows: ["Update Snapshots"]
varieties:
- accomplished
jobs:
check:
timeout-minutes: 60
runs-on: ubuntu-newest
steps:
- makes use of: actions/checkout@v2
- makes use of: actions/setup-node@v2
with:
node-version: "14.x"
- title: Set up dependencies
run: npm set up
- title: Set up Playwright Browsers
run: npx playwright set up --with-deps chromium
+
+ - title: Watch for Netlify Deploy
+ makes use of: mmazzarolo/wait-for-netlify-motion@8a7a8d8cf5b313c916d805b76cc498380062d268
+ id: get-netlify-preview-url
+ with:
+ site_id: "YOUR_SITE_ID"
+ env:
+ NETLIFY_TOKEN: ${{ secrets and techniques.NETLIFY_TOKEN }}
+
+
+ - title: Run Playwright assessments
- run: npx playwright check
+ run: WEBSITE_URL=${{ steps.get-netlify-preview-url.outputs.url }} npx playwright check
- makes use of: actions/add-artifact@v2
if: failure()
with:
title: playwright-report
path: playwright-report/
retention-days: 30Lastly, we will replace the update-snapshots.yml workflow identical to we did above.
title: Replace Snapshots
on:
issue_comment:
varieties: [created]
jobs:
updatesnapshots:
if: ${{ github.occasion.problem.pull_request && github.occasion.remark.physique == '/replace-snapshots'}}
timeout-minutes: 60
runs-on: ubuntu-newest
steps:
- makes use of: actions/checkout@v2
with:
fetch-depth: 0
token: ${{ secrets and techniques.GITHUB_TOKEN }}
- title: Get SHA and department title
id: get-department-and-sha
run: |
sha_and_branch=$(
curl
-H 'authorization: Bearer ${{ secrets and techniques.GITHUB_TOKEN }}'
https://api.github.com/repos/${{ github.repository }}/pulls/${{ github.occasion.problem.quantity }}
| jq -r '.head.sha," ",.head.ref');
echo "::set-output title=sha::$(echo $sha_and_branch | reduce -d " " -f 1)";
echo "::set-output title=department::$(echo $sha_and_branch | reduce -d " " -f 2)"
- title: Fetch Department
run: git fetch
- title: Checkout Department
run: git checkout ${{ steps.get-department-and-sha.outputs.department }}
- makes use of: actions/setup-node@v2
with:
node-version: "14.x"
- title: Set up dependencies
run: npm set up
+
+ - title: Watch for Netlify Deploy
+ makes use of: mmazzarolo/wait-for-netlify-motion@8a7a8d8cf5b313c916d805b76cc498380062d268
+ id: get-netlify-preview-url
+ with:
+ site_id: "YOUR_SITE_ID"
+ commit_sha: ${{ steps.get-department-and-sha.outputs.sha }}
+ env:
+ NETLIFY_TOKEN: ${{ secrets and techniques.NETLIFY_TOKEN }}
- title: Set up Playwright browsers
run: npx playwright set up --with-deps chromium
+
- title: Replace snapshots
- run: npx playwright check --replace-snapshots --reporter=listing
+ run: WEBSITE_URL=${{ steps.get-netlify-preview-url.outputs.url }} npx playwright check --replace-snapshots --reporter=listing
- makes use of: stefanzweifel/git-auto-commit-motion@v4
with:
commit_message: "[CI] Replace Snapshots"That ought to do it. Any longer, our Playwright visible regression assessments will run in opposition to the deploy previews.
Conclusion
I hope this weblog publish gave you a strong basis for constructing your visible testing setup.
Just a few concepts on how one can enhance it additional:
- You’ll most likely need to check full display snapshots utilizing the
fullScreenparameter as an alternative of simply the seen viewport. And perhaps seize cell snapshots too. - In case your internet software hundreds components of the UI asynchronously (e.g., pictures, movies), you’ll most likely need to look forward to them to be loaded or omit them from the assessments.
- You possibly can prohibit the
/update-snapshotscommand in order that it may solely be invoked by the repository’s homeowners. - You possibly can set off the snapshots updating stream from a distinct supply (e.g., webhooks) as an alternative of counting on GitHub feedback.
- You possibly can retailer the snapshots on a third-party storage resolution.
- When you’re utilizing deploy previews, you may optimize the workflows by parallelizing the step ready for the preview hyperlink with the remainder of the workflow.
Acknowledgments
Throughout this weblog publish, I copy-pasted snippets and paragraphs from the Playwright and the Cypress “Funcional VS visible testing” documentation.
