refactor: simplify video encoder using mediabunny sources #242
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Replace custom VideoEncoder implementation with mediabunny's built-in CanvasSource and VideoSampleSource for cleaner, more maintainable code. ## Changes ### PlayableVideoEncoder Refactoring - Replace custom VideoEncoder + EncodedVideoPacketSource with mediabunny's CanvasSource and VideoSampleSource - Support two modes: 'canvas' (for skeleton viewer) and 'bitmap' (for human viewer) - Use CanvasSource for direct canvas recording with automatic frame capture - Use VideoSampleSource for bitmap-based recording from ImageBitmaps - Simplify frame addition with automatic timestamp/duration calculation - Remove frame buffering logic (handled by mediabunny) ### Skeleton Viewer Updates - Pass canvas element to enable CanvasSource mode - Direct canvas capture for improved performance ### Human Viewer Updates - Continue using bitmap mode with VideoSampleSource - Works seamlessly with pix2pix translated frames ## Benefits - Simpler, more maintainable code - Leverages mediabunny's optimized encoding pipeline - Better separation of concerns (canvas vs bitmap recording) - Automatic backpressure handling - Reduced custom encoding logic 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Remove canvas source support to minimize the PR scope. This change focuses on using only VideoSampleSource (bitmap mode) from mediabunny, making the refactor more incremental. Canvas source support can be added in a future enhancement once bitmap mode is proven stable. ## Changes - Remove CanvasSource import and usage - Remove EncoderMode type and mode branching logic - Simplify constructor to only accept ImageBitmap - Remove canvas parameters from pose viewer methods - Net reduction of 48 lines of code 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
H264 codec requires even-sized frames. When images have odd dimensions (e.g., 375x375), resize them to the nearest even dimensions (376x376) before creating VideoSamples. Fixes error: "The dimensions 375x375 are not supported for codec 'avc'; both width and height must be even numbers." ## Changes - Calculate even dimensions in createMP4Output() - Resize ImageBitmap to even dimensions in addFrame() if needed - Use high-quality resize for minimal quality loss - Clean up resized bitmap after use to prevent memory leaks 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Remove unused isPlayable() method and simplify code structure now that we only support MP4/H264. ## Changes - Remove unused `isPlayable()` method (was never called) - Remove `VideoCodec` type import (hardcode 'avc') - Remove `container` and `codec` instance properties (constants now) - Merge `createMP4Output()` logic into `init()` method - Hardcode MIME type to 'video/mp4' in finalize() - Keep `getMediaSourceClass()` (used by signed-language-output component) - Keep `isSupported()` (used for feature detection across codebase) Net result: Simpler, more focused encoder implementation optimized for H264/MP4 compatibility. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Simplifies the playable video encoder by replacing the custom VideoEncoder implementation with mediabunny's built-in CanvasSource and VideoSampleSource. This makes the code cleaner, more maintainable, and leverages mediabunny's optimized encoding pipeline.
Changes
PlayableVideoEncoder Refactoring
Skeleton Viewer Updates
addCacheFrame()to enable CanvasSource modeHuman Viewer Updates
Benefits
Technical Details
Before: Custom VideoEncoder manually handled encoding chunks, frame buffering, and packet management.
After: Mediabunny sources handle encoding automatically:
CanvasSource: For skeleton viewer - captures canvas state directlyVideoSampleSource: For human viewer - encodes ImageBitmap framesBoth modes use the same MP4/H.264 output format with QUALITY_HIGH bitrate for maximum compatibility.
Testing
🤖 Generated with Claude Code