Importing Siteimprove alfa generated EARL to WAI WCAG-EM Report tool fails

[BogdanCerovac]

I am trying to import EARL JSON-LD generated by Siteimprove alfa into WAI WCAG-EM report tool (https://github.com/w3c/wai-wcag-em-report-tool) but get an error and import fails.

After some debugging I discovered that Json file is missing the subject to pass the first parsing;

https://github.com/w3c/wai-wcag-em-report-tool/blob/836cefa8027099a7eab1e64fe6d99375933cee0e/src/stores/earl/assertionStore/index.js#L228

Before I do any changes to Siteimprove EARL generator I have to ask if there are any plans for synchronization of it or am I maybe forgetting something when exporting results to EARL using toEARL from https://github.com/Siteimprove/alfa/blob/11efd27b0e533271b1d4c17bf53dc22a0e44f0db/packages/alfa-earl/src/serializable.ts ?

[Jym77]

Thanks for spotting this. I'll look into it.
I guess we'll just need to add the correct property in the .toEARL function for Outcome 🤔
https://github.com/Siteimprove/alfa/blob/main/packages/alfa-act/src/outcome.ts#L45-L55

[BogdanCerovac]

Thank you for looking into it.

Yes, seems like adding .subject property to Outcome would pass on to next checks, yes.

[Jym77]

Looking more into it, it seems the subject has to be added manually.
The reason is that the individual outcomes (or rules) don't really know about the test subject, only the full audit knows the test subject (the page).

When I'm calling the CLI on one of ACT rules examples (node .\packages\alfa-cli\bin\alfa audit -f earl -o report.json --outcome passed https://act-rules.github.io/testcases/ff89c9/39c0a2b75b4795e2e3ebb46520c926c6e01c90bd.html), I get the following EARL bit:

{
    "@context": {
      "earl": "http://www.w3.org/ns/earl#"
    },
    "@type": "earl:Assertion",
    "earl:test": {
      "@id": "https://alfa.siteimprove.com/rules/sia-r16"
    },
    "earl:result": {
      "@type": "earl:TestResult",
      "earl:outcome": {
        "@id": "earl:passed"
      },
      "earl:info": "The element has all required states and properties",
      "earl:pointer": {
        "@context": {
          "ptr": "http://www.w3.org/2009/pointers#"
        },
        "@type": [
          "ptr:Pointer",
          "ptr:SinglePointer",
          "ptr:ExpressionPointer",
          "ptr:XPathPointer"
        ],
        "ptr:expression": "/html[1]/body[1]/div[1]"
      }
    },
    "earl:assertedBy": {
      "@id": "https://alfa.siteimprove.com/"
    },
    "earl:subject": {
      "@id": "https://act-rules.github.io/testcases/ff89c9/39c0a2b75b4795e2e3ebb46520c926c6e01c90bd.html"
    }
  }

Which contains the correct subject.

The relevant bit for adding the subject is done by the EARL formatter:

alfa/packages/alfa-formatter-earl/src/earl.ts

Lines 57 to 59 in db3bbff

	earl["earl:subject"] = {
	"@id": _["@id"],
	};

which is called from the CLI command:

alfa/packages/alfa-cli/bin/alfa/command/audit/run.ts

Line 111 in db3bbff

const output = formatter.get()(page, rules, outcomes);

You probably want to mimic that in your own script to correctly fill the subject.

[mgifford]

@BogdanCerovac & @Jym77 has there been any advances with this?

[Jym77]

Alfa can't really know the subject (the page URL) from the Audit alone, it needs to be provided to it somehow.
When the CLI is used, an URL has been provided, so that the subject can be filled in automatically, but when Audit.of() is called directly, it can't be done.

Consider the following case:

const page = // Build some page directly in Alfa, or read some pre-serialised page 
  // for example
  Page.from(JSON.parse(fs.readFileSync("page.json", "utf-8"))
);

const audit = Audit.of(page, rules)

What should be used to fill in the subject in such a case? What if the page is build directly in JSX in the test file? (we could use page.request.url but that not ideal)

It would be even harder to generate a decent subject when individual rules are called on small input directly (typically what we do in our own test suit). In such cases, we have an Outcome where there is nothing good to use as subject. So Outcome.toEARL() cannot really fill in a decent subject and we let callers fill it in (like it's done in the EARL formatter, which does use the input's @id but is called in a context where we know we have one).

[BogdanCerovac]

@mgifford - as @Jym77 described - it's on us to provide subject based on the scope of audit etc. as we can use Alfa audits on variety of subjects.

@Jym77 - when page.request.url is available it could maybe serve as a "smart default"?
Could Alfa maybe default to that URL when it is available?

[Jym77]

If I remember correctly, it is always available and defaults to about: blank when unknown.
I'm uncomfortable in copying the "default value" logic in an other place to reverse it and guess whether what we read is the default value or not.
Additionally, this would create wrong results in cases where the subject is a set of pages and each of them has a perfectly fine URL. This logic would break the subject into several which is likely unwanted behaviour.

[BogdanCerovac]

@Jym77 - gotcha, totally understand, thanks.