Failures on DASH-IF Validator

[andyburras]

I tried running a couple of the test content media through the DASH-IF Validator (from https://github.com/Dash-Industry-Forum/DASH-IF-Conformance ) but both gave errors.

Looks like the audio is failing to validate at all.

The video gave:

avc_sets/12.5_25_50/t1/2022-01-17 and avc_sets/12.5_25_50/t1-cenc/2022-01-17:

Adaptation Set 0
error###        Failed to process Adaptation Set 0, Representation 0!

Adaptation Set 1
../src/ValidateBitStreams.cpp : 2038 : Warning: Validate_NAL_Unit: more than 8 (11) trailing bits
../src/ValidateAtoms.cpp : 4000 : colr atom of type nclx found, the software does not handle colr atoms of this type. 
../src/ValidateAtomList.cpp : 2331 : WARNING: In moov-1:udta-1 - unknown/unexpected atom 'meta'
### error: styp-2 
###        ../src/ValidateAtomList.cpp : 1362 : ### error: styp-2 
###        styp not at the begining of a segment (abs. file offset 1370979), this is unexpected
Program error! Requested infomration is at offset 36333262, which is in a removed region at index 28 (offset: 35185047, removed size: 1313526), exiting!profiling:/home/rodolphe:Cannot create directory

The video reports the errors when 'Segment Validation', 'DASH-IF', and 'CMAF' are selected.

[rbouqueau]

I think, on the latest master branch of the SegmentValidator that should be integrated soon in the DASH Conformance Software, that the "Program error!" is fixed. However the erroneous checks (e.g. "styp not at the begining of a segment") need to be addressed separately.

And also AFAIU these errors also went away on the latest version of the content. @nicholas-fr Do you confirm? If so, I suggest closing this issue on this repository.

[Lis-TV]

The player is using MediaSource API.
So neither MPD nor media segment does not need to conform MPEG-DASH spec.

[jpiesing]

The player is using MediaSource API.
So neither MPD nor media segment does not need to conform MPEG-DASH spec.

The test content will have an existence outside the device playback tests so needs to be valid.
Anything not valid risks causing confusion about whether it's breaking things or whether it's harmless.

[gitwjr]

2022-06-12 meeting: @rbouqueau @nicholas-fr
Romain will rerun the test on the content after the latest content and JCCP are merged. Nicholas will post an XML health check issue with DASH-IF on  the validator referring to this issue regarding an error he is seeing.

[nicholas-fr]

After re-running the validation script I see JCCP validator failures in the current (2022-10-17) test content, which need to be addressed/explained when re-generating the content.
Full logs attached: test-content-validation_2023-03-16.zip

Issues:

{
"spec": "CMAF",
"section": "Section 7.3.2.2",
"test": "The concatenation of a CMAF Header and all CMAF Fragments in the CMAF Track in consecutive decode order SHALL be a valid fragmented ISOBMFF file",
"messages": [
    "Representation 1 not valid"
],
"state": "FAIL"
}

{
"spec": "CMAF",
"section": "Section 7.3.2.4",
"test": "Each CMAF Fragment in combination with its associated Header SHALL contain sufficient metadata to be decoded and displayed when independently accessed",
"messages": [
    "Width found for representation / track 1",
    "Height found for representation / track 1",
    "'profile_idc' found for representation / track 1",
    "'level_idc' found for representation / track 1",
    "FPS info not found for representation / track 1"
],
"state": "FAIL"
}

{
"spec": "CMAF",
"section": "Section A.2",
"test": "For a CMAF Track to comply with one of the media profiles in Table A.1, it SHALL conform to the colour_primaries, transfer_characteristics and matrix_coefficients values from the options listed in the table",
"messages": [
    "Valid color primaries found",
    "Valid transfer characteristics found",
    "FAIL"
],
"state": "PASS"
}

Note: The above failure seems to be a validator problem (raised here Dash-Industry-Forum/DASH-IF-Conformance#649).

{
"spec": "CTAWAVE",
"section": "WAVE Content Spec 2018Ed-Section 4.1",
"test": "WAVE content SHALL include one or more Switching Sets conforming to at least one WAVE approved CMAF Media Profile",
"messages": [
    "Switching Set 0 found with only tracks conforming to Media Profile H",
    "Video track found, but WAVE video track missing"
],
"state": "FAIL"
},
{
"spec": "CTAWAVE",
"section": "WAVE Content Spec 2018Ed-Section 5",
"test": "If a video track is included, then conforming Presentation will at least include that video in a CMAF SwSet conforming to required AVC (HD) Media Profile",
"messages": [
    "Switching set not found"
],
"state": "FAIL"
}

"Period": {
"verdict": "FAIL",
"info": [
    "Stream found to conform to a CMAF Presentation Profile: "
],
"test": [
{
"spec": "WAVE Content Spec 2018Ed",
"section": "Section 6.2",
"test": "WAVE CMFHD Baseline Program Shall contain a sequence of one or more CMAF Presentations conforming to CMAF CMFHD profile",
"messages": [
    "Not all CMAF Switching sets are CMFHD conformant"
],
"state": "FAIL"
    }
]
}

Issues that are t2/t16 specific (multiple chunks per fragment):

"spec": "CMAF",
"section": "Section 7.5.17",
"test": "For 'trun' version 1, the composition time of 1st presented sample in a CMAF Segment SHALL be same as first Sample decode time",
"messages": [
    "Representation 1 valid",
    "Representation 1 not valid",
    "Representation 1 not valid",
    "Representation 1 not valid",
    "Representation 1 not valid",
    "Representation 1 not valid",
    "Representation 1 not valid",
    "Representation 1 not valid",
    "Representation 1 not valid",
    "Representation 1 not valid",
    "Representation 1 not valid",
    "Representation 1 not valid",
    "Representation 1 not valid",
    "Representation 1 not valid",
    "Representation 1 not valid",
[...]
    "Representation 1 valid",
    "Representation 1 not valid",
    "Representation 1 not valid",
    "Representation 1 not valid",
    "Representation 1 not valid",
    "Representation 1 not valid",
    "Representation 1 not valid",
    "Representation 1 not valid",
    "Representation 1 not valid",
    "Representation 1 not valid",
    "Representation 1 not valid",
    "Representation 1 not valid",
    "Representation 1 not valid",
    "Representation 1 not valid",
    "Representation 1 not valid"

],
"state": "FAIL"
}

[rbouqueau]

I think that the version implemented in the validator is deprecated. The latest spec (dpctf v1.39) seems to take the edit lists into account (5.3.2.4):

An edit list EL[k] that may be present in the CMAF header describing the difference between the composition time and the presentation for this track in the CMAF Presentation.

[jpiesing]

I think that the version implemented in the validator is deprecated. The latest spec (dpctf v1.39) seems to take the edit lists into account (5.3.2.4):

Just because edit lists weren't explicitly mentioned in earlier versions doesn't mean they were excluded. In theory anything in ISOBMFF could be present.

[rbouqueau]

I'm out of our JCCP internal call to better understand the problem.

After the JCCP conformance tool refactoring, new error messages started to be reported. New messages (both fails and successes in fact) started to appear because new checks were executed (which is good). These checks were not executed because an early-exit-on-error check. This check was removed during the refactoring because some partial checks could still be conducted. In addition the main cause of the early-exit-on-error was ISOSegmentValidator crashes. Both causes (crashes and early-exit) explain the appearance of these new checks.

Now we face a new situation with these new error messages:

1. Errors come from JCCP. Then the JCCP team will fix them. We will set up the old conformance checker again so that people can check if the error belongs to this category. There are errors in the grey area that the JCCP team is willing to investigate (e.g. the byte-range caching broken mechanism - even though the old validator shows the error) . The JCCP team reasoning here is that there are probably low-hanging fruits in broken checks at an infrastructure level that would help maintain a high level of confidence in the conformance tool.
2. Option2: these conformance errors are real. Erik (main dev for JCCP) mentioned early that the "old" validator would return some false positive. I think we may have underestimated the frequency of errors in these unexecuted checks. Some messages are unclear and deserve some investigation.

• Some messages don't make any sense today. They come from outdated specs (e.g. the WAVE implementation from 2018). Sometimes specs breaks backward compatibility (e.g. dpctf spec last year), this is something that was discussed and which requires more coordinate conformance. Specs in the conformance tool also need to be kept up-to-date.

I also have looked at each message individually. For the me the content generated on 2022-10-17 and 2023-04-06 is safe.

[jpiesing]

To be clear, is there any alternative to WAVE having a have a list of false JCCP errors that are to be ignored and each time the content is updated or re-generated, the list of JCCP errors will have to be checked against this list of false errors and 'known' false errors discarded?

[rbouqueau]

Maybe this kind of mechanism should be on the conformance tool side. What do you think?

ATM the JCCP team is finalizing the tests. Then a JCCP release should follow. I expect it should improve the situation already, but I don't know by how much.

[rbouqueau]

NB: https://github.com/Dash-Industry-Forum/DASH-IF-Conformance/releases/tag/v2.2.1. There are still a few wrong negatives though.

[jpiesing]

NB: https://github.com/Dash-Industry-Forum/DASH-IF-Conformance/releases/tag/v2.2.1. There are still a few wrong negatives though.

• What is WAVE supposed to do about this? Remember the correct set of wrong negatives and, each time it runs, check for any additional negatives?
• Do github issues exist for each of the wrong negatives in the JCCP repo?

[rbouqueau]

I don't think there anything to do on the WAVE side, except for using the latest version of the conformance tool and waiting for the result of the on-going discussions on the JCCP side. The WAVE test vectors have been very useful for the JCCP validation already.

Do github issues exist for each of the wrong negatives in the JCCP repo?

That's probably what's going to happen. There are still discussions going on about the resolution. The conformance tool has no official maintainer at the moment.