Add a flag to enforce/guarantee we always get rVFC callback?

[lincolnneu]

I understand making rVFC callback best effort is to not affect video performance and user experience. But this design decision makes the problem become super complicated if we want to rely on rVFC to get accurate PTS timestamp: we have to implement a time calibration onSeeked or onPaused, in order to get the PTS of frame currently displayed when video is paused.

I beg the chrome developer to add a flag in rVFC to enforce if we always get the callback, regardless of whether main thread is busy or whatever reason, just like we'll always get an onSeeked event when it's seeked. It's better to let developer to decide if we should compromise callback reliability to video performance. E.g. we can give in the callback guarantee during video is playing, but we want to enforce the callback is fired when video is paused or when we seek to a timestamp, so the PTS time we have is not stale. In this way we can build a frame aware video player easily.

[lincolnneu]

Or rVFC makes sure to always fire to tell us PTS time. The expensive part of the callback can be compromised (such as paint to canvas). We just need a reliable way to get PTS time (I guess this is not expensive?).

[dalecurtis]

This isn't possible unfortunately; if it was we'd always have the flag on :)

When paused you're always going to get the right frame callback after a seek. The best effort only applies to during playback. Due to the threading involved, it's not possible to provide a guarantee that you are getting the information for the frame on screen.

[lincolnneu]

To double check in #66 we reach to a conclusion that:

The latest frame can internally be updated on the compositor thread after we read its metadata, so there might be a mismatch.

This does not affect

When paused you're always going to get the right frame callback after a seek.

Right?

[dalecurtis]

Correct, pause is unaffected since only 1 frame is ever rendered in the pause case (until play() is called anyways).

[lincolnneu]

In today's discussion on #59 we get some new ideas. Post here to keep the discussion:

If you are paused, and the pause has completed and the internal state stabilized, and then you seek, that statement is true.

We'll do the seek to another frame in the onpause() event (currentTime is T). On the onseeked() event we seek back to the time when we pause (T). In this way we convert the play->pause to a pause->seek->seek. Can we get the right callback for T using this strategy?

[tguilbert-google]

The best way would be to try it out with a timestamped video and see if it works in practice.

If you chain seeks without waiting for rVFC callbacks between, you could have a frame/metadata mismatch. E.g. seek#1 to Frame A queues an rVFC callback to be run in the next rendering steps, seek#2 causes the currentTime to match frame B's. During the rendering steps, we fire the callback with Frame A's metadata, at the same time the current frame is updated from A to B.

It possible this doesn't happen in practice, but there is no guarantee against it in the code.

[lincolnneu]

Yeah I've tested this thousands of times against timestamped video and the mechanism works well (using <video/> instead of mirroring the image data to </canvas>. We only care about the pts timestamp matches the eventual frame the video player shows when paused). I just want to have a theoretical proof.

latestSeekTime <= metadata.presentationTime can catch the corner case you mentioned. In the onseeked event of #2, since we're calling the rVFC of #1 seek, the metadata.presentationTime will < the performance.now() of when we perform seek #2.

Is there any other corner case?

[tguilbert-google]

It's hard to offer a theoretical proof... The seek/pause events happen in the Blink/HTML layer, while the rVFC events come from further down the stack, on a different thread, at a different moment of the event loop processing model... So it's hard to reason about end to end.

IDK if we already covered this corner case, but:
You could have an rVFC callback where latestSeekTime <= metadata.presentationTime is true (in the rVFC for frame A from seek#1), just as the underlying frame is updated (to frame B from seek#2), and you might never get an rVFC callback for that 2nd seek/frame B.

Other than that, I don't have anything else coming to mind, but I could still be missing something.

If your approach works well, then it's probably already solid enough.

[lincolnneu]

you might never get an rVFC callback for that 2nd seek/frame B.

We'll also check if calibrationTime === videoNode.currentTime in the onSeeked event. So if we don't get rVFC for #2 seek, we'll trigger another round of time calibration. Usually it'll be resolved after 1 or 2 rounds, though very rare.

[lincolnneu]

Just come up with this idea to address the crazy corner case in seek after pause:

If any of the step fails the assertion(expect), redo the time calibration process. In this way we can catch the case where underlying frames update without rVFC for seek#2.

[lincolnneu]

The seek/pause events happen in the Blink/HTML layer, while the rVFC events come from further down the stack, on a different thread, at a different moment of the event loop processing model

Is <video/>.currentTime also in the Blink/HTML layer? How can <video/>.currentTime make sure the currentTime is accurate despite the multithreading?

[tguilbert-google]

The currentTime is dictated by the audio's clock on the main thread. I don't think there's risk of inaccuracy here.

[lincolnneu]

The issue of videoNode.currentTime is the same timestamp might be interpreted as different frame (+-1 frame) in ffmpeg than chrome, which is caused by rounding errors and different precision levels used by chrome and ffmpeg.

The PTS timestamp from rVFC is a great solution to this frame accuracy problem, if we can have the guarantee asked in this thread or we have strategy to calibrate time.

[lincolnneu]

Updated experiment result: pause then chain seek->rvfc->onseeked and make sure every onseeked the rvfc is updated does work, but the user visual experience is pretty bad (sometimes can see +-1 frame switching quickly for quite a while).

Seeking too soon/frequently will cause onseeked happens before rvfc. I optimize it by performing time calibration after pause every 500 ms until it converges, to give browser enough time to stabilize and then seek. Also will show a suspense/spinner during this process to hide this visual confusion.

Experiment detail:

Test video: 25 fps constant frame rate timestamped video.
Evaluation frame accuracy method: Math.floor(videoNode.currentTime * 25) === Math.round(metadata.mediaTime * 25)

1. Seek to random time. Evaluate frame accuracy.
2. Set videoNode.playbackRate (0.07-5) to random value (to simulate videos with different frame rate).
3. Play the video, wait random time, then pause.
4. Once the time calibration indicator (spinner) disappears, evaluate frame accuracy.
5. Repeat step 1-4 5 times

Result:

Run the test 1000 times. 996 passed, 4 failed. The 4 failed cases are for the same reason: when videoNode.currentTime is 599.88s, chrome video player displays frame 14996 (and so our rvfc + time calibration believes it's 14996), but Math.floor(599.88 * 25) === 14997.

The 4 false negative results prove that PTS timestamp is more reliable to determine frame compared to videoNode.currentTime.

The above experiment also proves rVFC mediaTime is accurate (at least for the videos encoded in the same way as the test video) when performing simple seek (video is not playing, or video is paused and stabilized).