Wednesday, September 28, 2022
HomeJavaScriptAutomating visible UI assessments with Playwright and GitHub Actions · mmazzarolo.com

Automating visible UI assessments with Playwright and GitHub Actions · mmazzarolo.com


Visible UI testing (often known as regression testing) is a testing method to validate that your modifications don’t have any sudden influence on the UI.
Sometimes, such assessments take a picture snapshot of your entire software below check or a selected aspect after which examine the picture to a beforehand authorised baseline picture. If the pictures are the identical (inside a set pixel tolerance), it’s decided that the online software seems the identical to the consumer. If there are variations, then there was some change to the DOM format, fonts, colours, or different visible properties that have to be investigated.

This publish will discover how you can automate visible regression assessments of a “fashionable” internet software utilizing Playwright and GitHub actions. The aim is to construct a testing setup that checks for UI regressions on every pull request and permits selectively updating the baseline pictures when wanted.

Primary data of JavaScript (or TypeScript) and GitHub Actions is beneficial.

We assume you’ll combine this setup into an current internet software. If you wish to strive it from scratch, I like to recommend scaffolding a brand new internet software utilizing Vite.

You will discover a whole instance of the ensuing software in this GitHub repo.

For reference, right here’s what the tiny internet software we’re testing seems like:



website

Playwright setup

For our testing setup, we’ll use Playwright, an E2E testing framework. I like Playwright as a result of it offers an ideal developer expertise and good defaults out of the field, however you may obtain the identical consequence with comparable instruments corresponding to Cypress or Selenium.

To set up Playwright cd into our venture and run the next command:

npm init playwright

We’ll be prompted with a number of totally different choices (e.g. JavaScript vs TypeScript, and so forth.). When requested, we should always add a GitHub Actions workflow to run our assessments on CI simply:

✔ Add a GitHub Actions workflow? (y/N) · true

As soon as achieved, Playwright will add an instance check in ./assessments/instance.spec.ts, an instance E2E file in ./tests-examples/demo-todo-app.spec.ts (that we will ignore), and the Playwright configuration file in ./playwright.config.ts.

✔ Success! Created a Playwright Take a look at venture at ~/your-project-dir

Inside that listing, you may run a number of instructions:

  npx playwright check
    Runs the end-to-end assessments.

  npx playwright check --project=chromium
    Runs the assessments solely on Desktop Chrome.

  npx playwright check instance
    Runs the assessments in a selected file.

  npx playwright check --debug
    Runs the assessments in debug mode.

  npx playwright codegen
    Auto generate assessments with Codegen.

We advise that you simply start by typing:

    npx playwright check

And take a look at the next recordsdata:
  - ./assessments/instance.spec.ts - Instance end-to-end check
  - ./tests-examples/demo-todo-app.spec.ts - Demo Todo App end-to-end assessments
  - ./playwright.config.ts - Playwright Take a look at configuration

Go to https://playwright.dev/docs/intro for extra info. ✨

The Playwright configuration file generated by the scaffolding course of (./playwright.config.ts) offers some good defaults, however I like to recommend making use of a few modifications.

First, to simplify our setup, replace the initiatives listing to run our assessments solely on Chromium:


initiatives: [
  {
    name: "chromium",
    use: {
      ...devices["Desktop Chrome"],
    },
  },

We are able to replace the venture listing in a while, to incorporate extra browsers/units, if wanted.

Then, configure the webServer part in order that Playwright can robotically serve our software once we run our assessments:


webServer: {
  command: 'npm run dev --port 8080',
  port: 8080,
  reuseExistingServer: true
},

Generate the preliminary snapshots

The instance check generated by Playwright (./assessments/instance.spec.ts) will not be a visible regression check, so let’s delete its content material and add a tiny check that fits our wants:

import { check, anticipate } from "@playwright/check";

check("instance check", async ({ web page }) => {
  await web page.goto("https://mmazzarolo.com/"); 
  await anticipate(web page).toHaveScreenshot();
});

For visible regression testing, Playwright consists of the power to provide and visually examine snapshots utilizing await anticipate(web page).toHaveScreenshot(). On first execution, Playwright check will generate reference snapshots. Subsequent runs will examine in opposition to the reference.

We’re lastly able to run your check:

The check runner will say one thing alongside the road of:

Error: instance.spec.ts-snapshots/example-test-1-chromium-darwin.png is lacking in snapshots, writing precise.

That’s as a result of there was no baseline picture but, so this methodology took a bunch of snapshots till two consecutive snapshots matched, and saved the final snapshot to file system. It’s now able to be added to the repository (in assessments/instance.spec.ts-snapshots/example-test-1-chromium-darwin.png):



The image generated by our first test run
The picture generated by our first check run

Now, if we run our check once more:

The check ought to now succeed as a result of the present UI matches the UI of the reference snapshot generated within the earlier run.

Replace the snapshots domestically

Let’s transfer to the fascinating half.

If we make any visible change to the UI and rerun our assessments, they may fail, and Playwright will present us a pleasant diff between the “precise” and “anticipated” snapshot:

In instances corresponding to this one, once we need to make some voluntary modifications to a web page, we have to replace the reference snapshots. We are able to do that with the --update-snapshots flag:

npx playwright check --update-snapshots
[chromium] › instance.spec.ts:3:1 › instance check
assessments/instance.spec.ts-snapshots/example-test-1-chromium-darwin.png is re-generated, writing precise.

Now that we’ve seen how you can run visible assessments and replace the snapshots domestically, we’re prepared to maneuver to the CI stream.

Operating the assessments in CI utilizing GitHub Actions

GitHub Actions is a steady integration and steady supply (CI/CD) platform that permits you to automate your construct, check, and deployment pipeline. With GitHub Actions we will create workflows that construct and check each pull request to a repository.

A great place to begin for a visible regression testing setup is to run the assessments on every pull request creation and replace. Fortunately, Playwright already generated a useful GitHub Actions workflow to run assessments for this particular use case in .github/workflows/playwright.yml.
This workflow ought to work completely high-quality out-of-the-box, however I like to recommend tweaking it a bit to:

  1. Set up solely the browsers we’re focusing on with our assessments (--with deps chromium);
  2. Save the check consequence artifacts solely when assessments fail to keep away from storing pointless artifacts (if: failure()).
 title: Playwright Assessments

 on:
   push:
     branches: [main, master]
   pull_request:
     branches: [main, master]

 jobs:
   check:
     timeout-minutes: 60
     runs-on: ubuntu-newest
     steps:
       
       - makes use of: actions/checkout@v2
       - makes use of: actions/setup-node@v2
         with:
           node-version: "14.x"
       - title: Set up dependencies
         run: npm set up
       - title: Set up Playwright Browsers
-        run: npx playwright set up --with-deps
+        run: npx playwright set up --with-deps chromium
       
       - title: Run Playwright assessments
         run: npx playwright check
       
       - makes use of: actions/add-artifact@v2
-        if: all the time()
+        if: failure()
         with:
           title: playwright-report
           path: playwright-report/
           retention-days: 30

As soon as we commit this workflow within the GitHub repo, submitting a brand new pull request will set off the Playwright assessments.

Nonetheless, our check will doubtless fail at this level as a result of the reference snapshots in our repo have been captured in an surroundings that differs from the one used on CI (until we created them on a machine operating Ubuntu).

For extra info on the check consequence, we will examine both the GitHub Motion output or the check artifacts:



github actions artifacts

For instance, if we generated the reference snapshots on macOS, we’ll obtain an error stating that the snapshots for Linux haven’t been discovered:

Error: assessments/instance.spec.ts-snapshots/example-test-1-chromium-linux.png is lacking in snapshots

Let’s see how we will replace the reference snapshots for the CI surroundings.

Producing the reference snapshots domestically utilizing --update-snapshots was a bit of cake, however on CI, it’s a distinct story as a result of we should determine the place, how, and when to retailer them.

There are many methods we will deal with this stream, however for the sake of simplicity, let’s begin easy.
One sample that has labored nicely for me is to make use of a GitHub Motion workflow to replace the reference snapshots when a selected remark with a “/update-snapshots” textual content is posted in a pull request.
The thought is that each time we submit a pull request that we anticipate to influence the UI, we will publish the “/update-snapshots” remark in order that CI will generate and commit the up to date snapshots within the pull request department.













title: Replace Snapshots

on:
  
  
  
  
  issue_comment:
    varieties: [created]

jobs:
  updatesnapshots:
    
    
    if: ${{ github.occasion.problem.pull_request && github.occasion.remark.physique == '/replace-snapshots'}}
    timeout-minutes: 60
    runs-on: ubuntu-newest
    steps:
      
      - makes use of: actions/checkout@v2
        with:
          fetch-depth: 0 
          token: ${{ secrets and techniques.GITHUB_TOKEN }}
      
      
      
      - title: Get SHA and department title
        id: get-department-and-sha
        run: |
          sha_and_branch=$(
            curl 
              -H 'authorization: Bearer ${{ secrets and techniques.GITHUB_TOKEN }}' 
              https://api.github.com/repos/${{ github.repository }}/pulls/${{ github.occasion.problem.quantity }} 
            | jq -r '.head.sha," ",.head.ref');
          echo "::set-output title=sha::$(echo $sha_and_branch | reduce -d " " -f 1)";
          echo "::set-output title=department::$(echo $sha_and_branch | reduce -d " " -f 2)"
      
      - title: Fetch Department
        run: git fetch
      - title: Checkout Department
        run: git checkout ${{ steps.get-department-and-sha.outputs.department }}
      
      - makes use of: actions/setup-node@v2
        with:
          node-version: "14.x"
      - title: Set up dependencies
        run: npm set up
      - title: Set up Playwright browsers
        run: npx playwright set up --with-deps chromium
      
      - title: Replace snapshots
        run: npx playwright check --replace-snapshots --reporter=listing
      
      - makes use of: stefanzweifel/git-auto-commit-motion@v4
        with:
          commit_message: "[CI] Replace Snapshots"

If we publish this workflow, create a brand new pull request, and add a “/update-snapshots” remark, CI will deal with producing the reference snapshots.



update snapshots succes

Good!

That’s it — now we have totally automated visible regression testing on our Steady Integration. Each time new modifications are posted in new pull requests, our assessments will guarantee we’re not mistakenly making modifications to the UI. We are able to additionally replace our baseline snapshots by posting a remark within the pull request.

In case your venture helps preview hyperlinks (corresponding to those robotically generated by Netlify, Vercel, and so forth.), carry on studying. In any other case, soar to the Conclusion part under.

Run Playwright assessments in opposition to deploy preview (Netlify, Vercel, and so forth.)

In case your internet software runs on platforms corresponding to Netlify and Vercel, it’s doubtless you’re utilizing their “Deploy Preview” characteristic: a strategy to preview pull request modifications with out impacting the online software in manufacturing.
Deploy previews are enabled by default for GitHub pull requests, and so they work by deploying them to a novel URL totally different from the one your manufacturing website makes use of.

In case your internet software makes use of deploy previews, we will combine them into our visible regression testing workflows to match the snapshots in opposition to them as an alternative of the native internet server.
This method brings two advantages. First, we keep away from spinning up a neighborhood webserver only for operating the assessments. Second, by operating our assessments in opposition to a preview hyperlink, we’ll produce extra dependable snapshots as a result of preview hyperlinks are a 1:1 illustration of what we should always see in manufacturing.

From a high-level view, there are three important modifications we have to make in our codebase to run our Playwright assessments in opposition to deploy previews:

  1. Permit passing the URL to check as a parameter to make our assessments conscious of the deploy preview hyperlink.
  2. Replace our GitHub Motion workflows to attend for the deploy preview to be accomplished.
  3. Replace our GitHub Motion workflows to move to Playwright the deploy preview URL (as an surroundings variable).

Right here’s how we will obtain this in Netlify (when you’re utilizing Vercel or another platform that helps deploy preview, the modifications needs to be nearly similar).

First, replace the use.baseURL worth of playwright.config.ts to obtain the deploy URL as an surroundings variable (WEBSITE_URL):

use: {
  baseURL: course of.env.WEBSITE_URL,
}

Additionally, let’s disable Playwright’s internet server if a WEBSITE_URL surroundings variable is offered:

webServer: course of.env.WEBSITE_URL
  ? undefined
  : {
    command: "npm run dev --port 8080",
    port: 8080,
    reuseExistingServer: true,
  },

Then, replace our playwright.yaml workflow to run assessments in opposition to the deploy preview.
To attend for the Netlify deploy preview URL, we will use the mmazzarolo/wait-for-netlify-action GitHub Motion.

mmazzarolo/wait-for-netlify-action is a fork of probablyup/wait-for-netlify-action. By default, probablyup/wait-for-netlify-action assumes it’s operating inside a workflow triggered by a pull request push. In our case, the update-snapshots.yml workflow is triggered by a remark, so I forked this GitHub Motion to make sure it runs on any workflow, no matter what triggered it.

The wait-for-netlify-action GitHub Motion requires two issues:

  1. Setting a NETLIFY_TOKEN GitHub Motion secret with a Netlify Private Entry Token.
  2. Passing a Netlify Web site ID (from Netlify: Settings → Web site Particulars → Normal) to the site_id parameter within the workflow.
 title: Playwright Assessments

 on:
   push:
     branches: [main, master]
   pull_request:
     branches: [main, master]
   workflow_run:
     workflows: ["Update Snapshots"]
     varieties:
       - accomplished

 jobs:
   check:
     timeout-minutes: 60
     runs-on: ubuntu-newest
     steps:
       
       - makes use of: actions/checkout@v2
       - makes use of: actions/setup-node@v2
         with:
           node-version: "14.x"
       - title: Set up dependencies
         run: npm set up
       - title: Set up Playwright Browsers
         run: npx playwright set up --with-deps chromium
+      
+      - title: Watch for Netlify Deploy
+        makes use of: mmazzarolo/wait-for-netlify-motion@8a7a8d8cf5b313c916d805b76cc498380062d268
+        id: get-netlify-preview-url
+        with:
+          site_id: "YOUR_SITE_ID"
+        env:
+          NETLIFY_TOKEN: ${{ secrets and techniques.NETLIFY_TOKEN }}
+      
+      
+      - title: Run Playwright assessments
-        run: npx playwright check
+        run: WEBSITE_URL=${{ steps.get-netlify-preview-url.outputs.url }} npx playwright check
       
       - makes use of: actions/add-artifact@v2
         if: failure()
         with:
           title: playwright-report
           path: playwright-report/
           retention-days: 30

Lastly, we will replace the update-snapshots.yml workflow identical to we did above.

 title: Replace Snapshots

 on:
   
   
   
   issue_comment:
     varieties: [created]

 jobs:
   updatesnapshots:
     
     
     if: ${{ github.occasion.problem.pull_request && github.occasion.remark.physique == '/replace-snapshots'}}
     timeout-minutes: 60
     runs-on: ubuntu-newest
     steps:
       
       - makes use of: actions/checkout@v2
         with:
           fetch-depth: 0 
           token: ${{ secrets and techniques.GITHUB_TOKEN }}
       
       
       
       - title: Get SHA and department title
         id: get-department-and-sha
         run: |
           sha_and_branch=$(
             curl 
               -H 'authorization: Bearer ${{ secrets and techniques.GITHUB_TOKEN }}' 
               https://api.github.com/repos/${{ github.repository }}/pulls/${{ github.occasion.problem.quantity }} 
             | jq -r '.head.sha," ",.head.ref');
           echo "::set-output title=sha::$(echo $sha_and_branch | reduce -d " " -f 1)";
           echo "::set-output title=department::$(echo $sha_and_branch | reduce -d " " -f 2)"
       
       - title: Fetch Department
         run: git fetch
       - title: Checkout Department
         run: git checkout ${{ steps.get-department-and-sha.outputs.department }}
       
       - makes use of: actions/setup-node@v2
         with:
           node-version: "14.x"
       - title: Set up dependencies
         run: npm set up
+      
+      - title: Watch for Netlify Deploy
+        makes use of: mmazzarolo/wait-for-netlify-motion@8a7a8d8cf5b313c916d805b76cc498380062d268
+        id: get-netlify-preview-url
+        with:
+          site_id: "YOUR_SITE_ID"
+          commit_sha: ${{ steps.get-department-and-sha.outputs.sha }}
+        env:
+          NETLIFY_TOKEN: ${{ secrets and techniques.NETLIFY_TOKEN }}
       - title: Set up Playwright browsers
         run: npx playwright set up --with-deps chromium
+      
       - title: Replace snapshots
-        run: npx playwright check --replace-snapshots --reporter=listing
+        run: WEBSITE_URL=${{ steps.get-netlify-preview-url.outputs.url }} npx playwright check --replace-snapshots --reporter=listing
       
       - makes use of: stefanzweifel/git-auto-commit-motion@v4
         with:
           commit_message: "[CI] Replace Snapshots"

That ought to do it. Any longer, our Playwright visible regression assessments will run in opposition to the deploy previews.

Conclusion

I hope this weblog publish gave you a strong basis for constructing your visible testing setup.
Just a few concepts on how one can enhance it additional:

  • You’ll most likely need to check full display snapshots utilizing the fullScreen parameter as an alternative of simply the seen viewport. And perhaps seize cell snapshots too.
  • In case your internet software hundreds components of the UI asynchronously (e.g., pictures, movies), you’ll most likely need to look forward to them to be loaded or omit them from the assessments.
  • You possibly can prohibit the /update-snapshots command in order that it may solely be invoked by the repository’s homeowners.
  • You possibly can set off the snapshots updating stream from a distinct supply (e.g., webhooks) as an alternative of counting on GitHub feedback.
  • You possibly can retailer the snapshots on a third-party storage resolution.
  • When you’re utilizing deploy previews, you may optimize the workflows by parallelizing the step ready for the preview hyperlink with the remainder of the workflow.

Acknowledgments

Throughout this weblog publish, I copy-pasted snippets and paragraphs from the Playwright and the Cypress “Funcional VS visible testing” documentation.

RELATED ARTICLES

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Most Popular

Recent Comments