Customizing how the requirement verification status is summarized

[a-mahashabde]

Context
The overall verification status of a test run currently rolls up to "Pass with Findings" based on this documentation on tested requirements. However, there may be instances where in a particular implementation, the users may want to rollup/aggregate the summary verification status differently when findings are acceptable.

Question
Does InterUSS support customization of the requirements verification status? In particular, we currently see an overall verification status of "Pass with Findings" when a USS rejects attempts to plan over Accepted operational intents of a down USS (per ASTM F3548-21 SCD0005). Given that this is a conservative approach, we would like to change the overall verification status to "Pass" when there is a "Pass with Findings" low severity result for SCD0005 - "Warning (not a failure): planning got rejected, USS may have been more conservative".

[BenjaminPelletier]

There are two roll-up mappings: checks -> requirement and requirements -> overall. The linked table is the requirements -> overall mapping, and I don't think we've identified any changes to make for that roll-up. The first roll-up, however, has new functionality just recently added in #1352. The new configuration field is acceptable_findings within the definition of a tested_requirements artifact. As shown in the check -> requirements roll-up table, "Pass (with findings)" for a requirement can be converted into "Pass" by adding the failed check (for instance, the one producing the findings) to acceptable_findings. It sounds like the check producing findings you would like to accept is this one. To add this check to acceptable_findings, the FullyQualifiedCheck would be:

• scenario_type = scenarios.astm.utm.DownUSS
• test_case_name = "Plan Flight 1 in conflict with accepted operational intent managed by down USS"
• test_step_name = "Tested USS attempts to plan Flight 1"
• check_name = "Rejected planning"

Here is an example usage of this new field, and the effects of that usage can be seen in the CI report artifacts.

Regarding the findings produced by this particular check: the positive requirement in SCD0005 is for a USS to apply a particular priority value to op intents meeting certain criteria. The challenge is that InterUSS cannot directly observe the priority levels a USS has assigned to other op intents. This challenge is essentially fully resolved if/when a USS successfully plans over the op intent managed by a down USS -- this action demonstrates the USS must have assigned the prescribed lower priority to the existing op intent since it could not have planned over the existing op intent if it did not. However, the challenge remains if the USS rejects the requested op intent which intersects the existing op intent managed by a down USS -- this isn't a violation of a requirement, but InterUSS can no longer draw a strong conclusion regarding compliance with SCD0005 as the observed behavior is also consistent with the USS failing to comply with the requirement (e.g., assigning exactly the same priority to the op intent as if it were managed by a USS with normal status). Because there is one other check that, if failed, would indicate non-compliance to SCD0005 (if a USS crashes while attempting to plan in this situation, they can't have consistently applied SCD0005), merely not crashing when planning in the presence of an op intent managed by a down USS will result in SCD0005 being marked as verified. "Sufficiency" of testing is application/usage-dependent and cannot be objectively determined by InterUSS, but note that accepting failure of the Rejected planning check will show verification of compliance with SCD0005 even when InterUSS's primary test mechanism for SCD0005 is bypassed.

Please let me know if this new feature doesn't fully address your need, or if you have any other questions, clarifications, etc.

[github-actions]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.