Extend testdriver.js and wptrunner to test AAMs (platform accessibility APIs)#53733
Extend testdriver.js and wptrunner to test AAMs (platform accessibility APIs)#53733spectranaut wants to merge 38 commits intomasterfrom
Conversation
I wonder if we can make the name |
Amazing, thanks @jcsteh! I didn't know about that Firefox option, I always used the gnome accessibility trick. I changed the CLI arg to |
762b8de to
356e3c4
Compare
62fc8e5 to
b937113
Compare
5b98700 to
fbf3fdf
Compare
7c3363d to
0f4b380
Compare
This reverts commit d508e40.
Co-authored-by: Jonathan Lee <jonathan-j-lee@users.noreply.github.com>
550c19d to
c8c462e
Compare
spectranaut
left a comment
There was a problem hiding this comment.
@jonathan-j-lee thanks again for the review and sorry it took me so long to resolve them! I applied all your suggestions, just one open comment.
tools/wptrunner/wptrunner/executors/platformaccessibility/executorwindowsaccessibility.py
Outdated
Show resolved
Hide resolved
tools/wptrunner/wptrunner/executors/platformaccessibility/executoraxapi.py
Outdated
Show resolved
Hide resolved
tools/wptrunner/wptrunner/executors/platformaccessibility/executoratspi.py
Show resolved
Hide resolved
tools/wptrunner/wptrunner/executors/platformaccessibility/executorwindowsaccessibility.py
Show resolved
Hide resolved
tools/wptrunner/wptrunner/executors/platformaccessibility/ia2/constants.py
Outdated
Show resolved
Hide resolved
| if not valid_api_for_platform(api): | ||
| return "" | ||
|
|
||
| # self.impl will not exist if --enable-accessibility-api was not included | ||
| # or the necessary python requirements for accessibility API tests have not | ||
| # been installed. | ||
| if not self.impl: | ||
| subtests = len(test) if test else 1 | ||
| return ["Accessibility API testing not enabled."] * subtests |
There was a problem hiding this comment.
Good point on the throwing an error instead, I switched to using a custom error, see: 0b42a68
As for the other concern, about returning a list, it is just about readability of the test results. By sending back a list for each testable statement (if there are two), then you can see two assertions passed from the WPT test summary page. It is a bit of an awkward... but I'd like to iterate a bit on the design on the tests after confirmation that I can added accessibility API tests to WPT -- by extending the testdriver infrastructure in this way.
|
@jonathan-j-lee if you have time to look again, I opened a new PR with a different direction, different test type: #57696 It has all the changes you requested here. |
|
Closing in favor of: #57696 |
Closes web-platform-tests/interop-accessibility#210
This PR adds an extension to
testdriver.jsin order to test accessibility API, as defined by the ARIA, Core-AAM and HTML-AAM specifications. The RFC to extendtestdriver.jsin this way can be found here. The decision to open a PR with these changes was made at TPAC 2024.The change in this CL:
--enable-accessibility-api. So on regular runs of wpt.fyi, they will fail, for now, as you can see with:a. The option
--enable-accessibility-apihas been added. This option will install the appropriate python requirements for accessibility API tests and turn on accessibility for Firefox and Chromium browsers. If this option is not supplied, tests may fail withAccessibility API testing not enabled. The primary reason for the flag is that on linux, there are system requirements for the python requirements, so./wpt runwill fail in environments where the system requirements are not installed. For now I think it's better to only run the tests intentionally/behind a flag.a. The product name ("chrome", "firefox", etc) is passed back to the executor level. This is necessary because accessibility APIs are exposed externally to the browser, and we need to find the browser in the accessibility API by name. Ideally, this will be done by PID (to allow for parallelization), but that requires some changes to chromedriver and safaridriver.
executorplatformaccessibility.pyandtools/wptrunner/wptrunner/executors/platformaccessibility: these executors run tests against platform specific accessibility APIs, so there is an executor for Linux, Windows, and macOS. The Windows and macOS ones are proof of concepts and will need more work. The Linux one is fully featured and can run all of the tests added in this PR.To run tests on Linux:
On Debian distros:
apt install libatspi2.0-dev libcairo2-dev libgirepository1.0-devNotes about macOS and Windows
On Windows and macOS, even with the following dependencies, these tests will fail with the message "Fail: Test not implemented".
Thanks to @alice for the macOS code and thanks @jcsteh, the code to get IA2 interface was very inspired by the firefox test infrastructure: https://searchfox.org/mozilla-central/source/accessible/tests/browser/windows/a11y_setup.py
Running the tests on macOS
To run against Firefox, you need to turn on VoiceOver the screen reader.
To run against Chrome, you need to include --no-headless.