[Discovery] Choose new a11y checks tooling for CI

[jmbowman]

Some time ago we stopped running both pa11ycrawler and axe-core (via bok-choy Selenium tests) because they rarely told us anything useful and often broke for unrelated reasons. But we really do want some form of automated accessibility checks in CI, and both the available a11y tools and our web page rendering strategy have changed significantly since we first chose that previous generation of a11y checks. We should choose a new tool (or set of tools) that strikes the best balance between the following objectives:

• Runs many/most of the accessibility checks that can be usefully done in an automated manner
• Generates reports which provide useful information to professional a11y coordinators
• Minimal effort to set up to start producing useful results
• Few false positives that fail CI builds for spurious reasons
• Few failures for reasons unrelated to the accessibility of the tested code
• Resilient to changes in the source code which don't impact the accessibility of the rendered pages (for example, not closely tied to the current page layout expectations)
• Ideally open source (especially for integration into the openedx repositories that 2U doesn't manage)

Here is an incomplete list of potential tools to employ for this:

• axe-core (there are indications this may cease to be open source in the near future)
  • https://github.com/dequelabs/axe-core (also see Run axe-core/react in development as part of frontend-platform #39)
  • https://github.com/nickcolley/jest-axe
  • https://github.com/component-driven/cypress-axe
  • https://storybook.js.org/docs/react/writing-tests/accessibility-testing (for checking components, for example in Paragon)
• Lighthouse (uses axe-core under the hood, but gives fancier reporting)
  • https://github.com/GoogleChrome/lighthouse
  • https://unlighthouse.dev/
  • https://github.com/GoogleChrome/lighthouse-ci
  • https://github.com/mfrachet/cypress-audit
• IBM Equal Access Toolkit
  • https://www.ibm.com/able/toolkit/
  • https://www.ibm.com/able/toolkit/tools/#develop
  • https://github.com/IBMa/equal-access
• Pa11y
  • https://pa11y.org/
  • https://github.com/pa11y/pa11y-ci
• JSX-specific tooling
  • https://github.com/jsx-eslint/eslint-plugin-jsx-a11y
• Surveys of other options
  • https://www.w3.org/WAI/ER/tools/
  • https://a11y-automation.dev/automated-tools
  • https://www.digitala11y.com/open-source-accessibility-tools/

A/C:

• Discuss the list of objectives with Jeff Witt (2U's accessibility coordinator for edx.org) and add any that were overlooked
• Flesh out the list of potentially viable candidates
• Create a wiki page and/or spreadsheet with a rough estimation of how well each tool satisfies each of the stated objectives
• Suggest one or more tools to move forward with
• Write new tickets for next steps in employing the chosen tool(s)

[jmbowman]

Initial findings are outlined in https://docs.google.com/spreadsheets/d/1srZ2IInHZicE2QVukDFUkuoZBrynpDW3L0eLExuThJE/edit#gid=203478394 . After discussion with Jeff Witt:

• axe-core is still probably the best core engine to use
• We'd probably want to add the wcag22a and 2cag22aa tags to the axe-core config if initial testing goes well; see https://www.deque.com/axe/core-documentation/api-documentation/ for details.
• Lighthouse could work as a wrapper for axe-core, as long as it provides the ability to disable individual checks and add custom ones (like https://github.com/openedx/edx-custom-a11y-rules)
• Another UI option for axe-core reports is https://www.npmjs.com/package/axe-html-reporter
• We would ideally like to report on a larger set of checks than we actually treat as blockers to merging. Some checks are important to know about but can be fixed sometime after initial deployment.
• https://asqatasun.org/ and https://github.com/Siteimprove/alfa + https://github.com/Siteimprove/alfa-integrations may also be worth a look, although they're likely to be less mature. Asqatasun is interesting because it also checks for compliance with the French government's RGAA accessibility guidelines.

Potentially good candidate pages for testing the tools against:

• The new discussions panel in a course page (with at least 1-2 discussion posts present)
• Course home/outline page
• business.edx.org