accessibility testing with playwright

My last few projects at work I’ve been using Playwright for e2e testing instead of Cypress. Playwright is an open source tool from Microsoft, and is a fairly intuitive testing framework if you have any experience with other javascript testing libraries like jest, react-testing-library, mocha, chai, etc. It’s also pretty straightforward to integrate with Axe for automating some accessibility testing. Wonderful.

Here’s how I went about accessibility testing this website with Playwright.

setting up playwright

First, I installed playwright via npm init playwright@latest. This asks a few questions and generates a playwright.config.ts and some example tests. I am using typescript and wanted a github action workflow generated for me. Cool. I also needed to install the browsers that Playwright uses for its tests. I did not need to install any additional system deps, but you may need to (whether locally or in CI). The Playwright docs are generally pretty good, this is all outlined in them. Then I added a command to package.json, so I could run the tests easily.

    "test": "npx playwright test"

My personal laptop is getting up there in years, so despite installing versions of chromium, safari, and firefox… playwright actually won’t test webkit on my machine. So I altered the config to only run tests for chromium on my local and save testing the full suite of browsers for CI.

Once that was sorted out, Playwright was successfully running the example tests locally. Now to extend it for accessibility testing.

adding axe

Playwright’s docs cover the main part of this, but I’m going to paste the extension here for reference.

First npm i -d @axe-core/playwright to grab the playwright-compatible Axe library.

Then in a test-utils.ts file, we’re going to drop this in to create a re-usable fixture:

/* This file contains consistent configuration for a11y tests across other test files. */
import { test as base } from '@playwright/test'
import AxeBuilder from '@axe-core/playwright'

type AxeFixture = {
  makeAxeBuilder: () => AxeBuilder
}

// Extend Playwright's base test by providing "makeAxeBuilder"
//
// This new "test" can be used in multiple test files, and each of them will get
// a consistently configured AxeBuilder instance.
export const test = base.extend<AxeFixture>({
  makeAxeBuilder: async ({ page }, use, testInfo) => {
    const makeAxeBuilder = () =>
      new AxeBuilder({ page }).withTags([
        'wcag2a',
        'wcag2aa',
        'wcag21a',
        'wcag21aa',
      ])
    // .exclude('#commonly-reused-element-with-known-issue');

    await use(makeAxeBuilder)
  },
})

// Exported here for convenience
// so other test files can do import { test, export } from './utils'
export { expect } from '@playwright/test'

Now our test files can use makeAxeBuilder() to generate an axe-check with consistent configuration. I like to export {expect} from the utils file to keep my imports cleaner in the tests themselves. Personal preference.

testing

OK! With the base in place, we can now run accessibility tests against our site. A barebones test looks like this:

test('Home page renders without a11y errors', async ({
  page,
  makeAxeBuilder,
}) => {
  await page.goto('/')

  const accessibilityScanResults = await makeAxeBuilder().analyze()

  expect(accessibilityScanResults.violations.length).toEqual(0)
})

reporting

That isn’t super useful if there are errors, though. I want the results in an easy-to-read format so I can act on them. I’m doing that in two ways:

1. the axe-html-reporter package, which turns the results into a nicely structured HTML report I can view locally
2. massaging the results into a better json object that prints in the console.

This way, if it’s a simple error I can see it immediately in the console, but if there are a number of violations I can reference them easily with the HTML report.

Here’s how I set up my function to generate both reports. In my test-utils.ts file, I added this

import { createHtmlReport } from 'axe-html-reporter';
import fs from 'fs';

...

// Generate readable report outputs for a given check.
export const generateReport = (accessibilityScanResults, key) => {
  // axe-html-reporter builds a nice page, use that.
  const htmlReport = createHtmlReport({
    results: accessibilityScanResults,
    options: {
      projectKey: 'heffdotdev',
      doNotCreateReportFile: true
    },
  });

  // write report to file. test-results is gitignored
  const htmlReportDir = 'test-results/a11y'
  if (!fs.existsSync(htmlReportDir)) {
    fs.mkdirSync(htmlReportDir, { recursive: true })
  }
  fs.writeFileSync(`${htmlReportDir}/${key}.html`, htmlReport)

  // create useful json object
  const errors = accessibilityScanResults.violations.map(v => {
    return {
      issue: v.id,
      count: v.nodes.length,
      description: v.description,
      errors: v.nodes.map(n => {
        return {
          html: n.html,
          impact: n.impact,
          target: n.target,
          summary: n.failureSummary,
        }
      }),
    }
  })

  return {
    htmlReport,
    errors
  }
}

And in our barebones a11y test, we generate the reports if there are errors:

  if (accessibilityScanResults.violations.length > 0) {
    generateReport(accessibilityScanResults, 'homepage')
  }

We can now run the tests & get a readable, actionable report of any violations found.

conclusion

Speaking of found violations, boy did I find some! Running npm run test with barebones tests against my four main pages (homepage, blog, gallery, work) I was immediately hit with a number of violations per page. There was one obvious issue in the header: the menu icon that links to the site menu in the footer did not have an accessible name, because there was no text at all 🙈.

The remaining violations on most pages are related to contrast issues. The color schemes I came up with… they are not fully accessible. They pass the contrast check at most larger sizes, but smaller text fails. So I need to update my theme a bit. The emoji wall on the homepage also needs some attention.

Once I have the light and dark color schemes sorted out, I will be able to use Playwright to easily verify that they both pass across all the different page types. Neat!

You can view all of this site’s Playwright tests here.