how to verify / validate / approve audio test content

[jpiesing]

This arises from a discussion in CSTF about test content being submitted vs validated (or approved).

We take a lot of trouble to verify, validate & approve the video test content & to ensure that an independent pair of eyes is involved.

Audio test content is much less formal and thorough.

What needs to be done so that we can consider audio test content to be either approved or validated, whichever term you prefer.

[MichaelForsyth634]

On top of validating the generation process

we do the following checks on all audio media

The basic things we checked for in the audio

• check segments map to the MPD segment timeline
• check MPD passes dash if validator with CTA setting

After concatenating segments into a single mp4

• if encrypted using clearkey
  -- check MPD has correct kid (default kid 279926496a7f5d25da69f2b3b2799a7f)
  -- decrypt media (default key 00112233445566778899aabbccddeeff)
• check media plays (we use ffplay)
• check edit list box for delay value (should be present for most audio)
• use media info to check parameters match requested output

use ffmpeg to extract waveform

• ffmpeg -i audio.mp4 -c:a copy audio.aac
• ffmpeg -i audio.aac audio.wav

Then check audio.wav in waveform viewer

• padding at start should match edit list value
• waveform start after padding should match source audio [Note encoding will have distorted waveform so human judgement needed]

[rcottingham]

To add to the detail above, I’d like to clarify that the process for audio test content is thorough and formal. However, because it hasn’t always been visible within the WAVE/CSTF group, we understand why this perception may exist.

Here's how it worked for the currently approved audio content:

• Generation & Internal Validation: Audio test content was generated and validated internally by Resillion, with separate teams handling generation and validation.
• External Review: Validated content was shared with the media proponents (Dolby and Fraunhofer) for review and feedback.
• Iterative Fixes: Issues raised were addressed, and the cycle repeated until the content met requirements.
• Final Approval: Several rounds of iteration were completed before the audio content was approved by the proponents for inclusion.

While validation and approval were not conducted publicly within WAVE discussions, we hope the rigor applied was equivalent to that used for video content.

That said, we recognize the importance of transparency for group confidence. Going forward, we can improve visibility for the WAVE/CSTF group, for example by:

• Documenting validation steps and outcomes more clearly in an appropriate repo.
• Providing periodic updates in CSTF meetings when new audio content is under review.
• Recording approval status explicitly (e.g., “submitted,” “under validation,” “approved”) so that contributors and reviewers know where things stand.

---

To give a concrete example, here’s a snapshot of the validation rounds for the AAC streams, illustrating the process.:

• Initial (2023-03-16):
  • Full internal validation on all streams.
• Interim (2023-03-30):
  • Fixed streams 6, 7, 9, 10, 11, 12 following Fraunhofer feedback:
    • a) HE-AAC and HE-AACv2 streams did not have proper SAPs at all segment boundaries. At least at some the SBR (and PS) headers were missing;
    • b) On camc (streams 11 and 12) AudioChannelConfigurations in MPD updated to MC 5.1 (6ch)
• Final (2023-04-05):
  • Fixed HE-AAC and HE-AACv2 streams following Fraunhofer feedback:
    • SBR signaling changed from to “EXPLICIT_NOT_BC (explicit hierarchical)” to “EXPLICIT_BC”;
  • Resolved issue with offset being added to the time of segments introduced in the Interim delivery;
  • Corrected stream 10 codec (changed to HE-AACv2)
• Final v2 (2023-04-27):
  • Segments changed to 46 samples in length for HE-AAC and HE-AACv2