What is the expected timing of MSTP video frame enqueuing with other track events (configurationchange, applyConstraints promise resolution)

[youennf]

It would be nice for a web application to know when the settings of a track actually kicks in for enqueued VideoFrames in MSTP.

For some properties like size, VideoFrame object already has this info.
For plenty of settings, this is not the case (surface type, deviceId, all image settings)...

[youennf]

A first solution is to synchronise the events/promise resolution tasks with VideoFrame enqueuing tasks. Plus reusing the same task source.
That way MediaStreamTrack exposed settings apply to enqueued VideoFrames (as long as web page eagerly reads VideoFrames).

Another solution is to expose the settings in some other form.
For instance in VideoFrame directly (VideoFrameMetadata).
Or add a synchronisation mechanism between VideoFrame and MediaStreamTrack in case we do not want to expose all track settings in VideoFrameMetadata.

[jan-ivar]

MSTP has a buffer. How do you plan to contend with maxBufferSize?

I agree specifying timing is good. However, I don't think we can give any meaningful synchronous guarantees, given the async nature of ReadableStreams. I'm not sure we should either. If we couple things too tightly, I worry JS won't be able to add a transform into a pipe without breaking things.

[dontcallmedom-bot]

This issue was discussed in WebRTC December 2024 meeting – 10 December 2024 (Issue 115: What is the expected timing of MSTP video frame enqueuing with other track events)