Implement Faster Encoder/Decoder

[josephernest]

Congrats for the project!

Import

Adding a MP4 file is very slow: I tried with a 12 MB .mp4 file.

After 2 minutes I am still at "6.45% loading...", so it would require 30+ minutes to load a small 12 MB MP4 file!
What is in your opinion the reason of this slow processing?

Which browser API / algorithm do you use to import MP4 files ? What does it do exactly? Do you think there is room for improvement?
Maybe ffmpeg using WASM?

Suggestion: with this method: https://stackoverflow.com/questions/4429440/html5-display-video-inside-canvas
you can import a HTML5 <video> into a <canvas> with no loading time at all.

Export

Same question for export: which exporting/encoding technique do you use? Is there a ffmpeg or a browser API under the hood?

Or do you use a technique like https://webrtc.github.io/samples/src/content/capture/canvas-record/?
They use here the MediaRecorder API: https://github.com/webrtc/samples/blob/gh-pages/src/content/capture/canvas-record/js/main.js

Tested platform: Windows, Chrome 101

[bwasti]

Surprisingly, Chrome is much slower than other browsers on this front (from what I've found).

The approach used is extremely naive and using FFMpeg (or similar) is a good idea.

For import, I manually pull frames in one-by-one:

mebm/script.js

Lines 648 to 665 in a1c1316

	async convertToArrayBuffer() {
	this.video.pause();
	let d = this.video.duration;
	let name = this.name;
	for (let i = 0; i < d * fps; ++i) {
	let frame = await this.seek(i / fps);
	let sum = 0;
	for (let j = 0; j < frame.data.length; ++j) {
	sum += frame.data[j];
	}
	this.frames.push(frame);
	this.update_name((100 * i / (d * fps)).toFixed(2) + "%");
	}
	this.ready = true;
	this.video.remove();
	this.video = null;
	this.update_name(name);
	}

Export uses the native MediaRecorder as you mention (similar to frame-by-frame by it's native at least)

mebm/script.js

Line 1677 in a1c1316

const rec = new MediaRecorder(stream);

I am going to go ahead and relabel this as a feature request and look into compiling a simple encoder/decoder into a small wasm blob. I'd prefer to avoid full ffmpeg as it's quite large last I checked.

[josephernest]

Thanks for your answer!

For import, I manually pull frames in one-by-one:

Is it correct that you are playing the video, and storing each frame in RAM?

Naive question, but is it really required to store each frame in RAM?
I think it might not be mandatory:

• when you change the cursor timestamp and want to display the relevant frame (fixed image, not playing), you can just seek in the <video>, and ctx.drawImage(...) from the video element.
• when you want to play the video, then you can pull the frames at this time with https://stackoverflow.com/questions/4429440/html5-display-video-inside-canvas

Is it really mandatory to pull everything when adding the video?

Example: imagine the video is 5 minute long, there is no need for a 5-minute initial loading time. You can just pull the frames when they are needed for playback, just-in-time-style ;)

[bwasti]

You're 100% correct that a just-in-time solution would save a lot of compute here. I decided to use pure raw frames in RAM for a couple reasons

1. on Safari (primary testing), you can seek faster than playing the video so it wasn't a huge issue (unlike chrome, where seek seems to be slower than playing)
2. seek + display at playhead was presenting weird issues on some browsers (partial frames or just the wrong frames, I forget exactly)
3. if I wanted to edit the video at a lower level (e.g. editing colors or cropping in weird ways), I'd want a RAW layout as its a bit more intuitive to use

I think this whole thing deserves a refactor. Ideally it'd use a format that presents itself as "RAW" (individually addressable pixels at each frame) yet is actually stored in a compressed way.