New Feature: Green Screen Pipeline — AI keying, animated backgrounds, and auto-compositing

[calesthio]

Green Screen Pipeline is here

We just shipped the green screen pipeline — a complete system that takes raw green/blue screen talking-head footage and transforms it into a polished composite video with AI-powered keying, animated Remotion backgrounds, TikTok-style captions, and background music.

Record yourself in front of a green screen (even with uneven lighting), paste a prompt, and get back a finished video with your speaker composited over animated data visualizations, glassmorphism backgrounds, and professional captions.

---

What can you do with it?

Green screen talking head with data viz:

Take my green screen footage, remove the background, composite me over animated charts and stats that appear in sync with what I'm saying, and add TikTok-style captions.

Explainer video with animated backgrounds:

I have a talking-head recording on a green screen. Replace the background with an animated gradient mesh with floating orbs, overlay data visualizations that match my script, and export for YouTube.

Product demo with branded backgrounds:

Remove my green screen, put me over our branded Remotion template with animated stats and KPIs, add word-by-word captions, and mix in background music.

---

2 New Tools Built

Tool	What it does
green_screen_processor	End-to-end green/blue screen keying. Auto-detects screen color via histogram analysis. Tries FFmpeg chromakey first for speed, automatically falls back to rembg AI segmentation (U2Net) when color-based keying fails (uneven lighting, green spill on skin). Outputs clean alpha-keyed video.
green_screen_composite	Composites keyed speaker over any background (Remotion renders, static images, video). 4 layout presets: news_anchor (speaker 65% bottom, graphics shifted up — no overlap), full_behind (speaker fills frame, bg behind), pip (30% corner picture-in-picture), split (50/50 side by side). Alpha compositing via numpy with configurable thresholds.

---

The Green Screen Pipeline

raw green screen footage
  → green_screen_processor (auto-detect color → chromakey or rembg AI)
  → Remotion Explainer render (AnimatedBackground + data viz scenes)
  → green_screen_composite (layout: news_anchor / full_behind / pip / split)
  → remotion_caption_burn (TikTok-style word-by-word captions)
  → audio_mixer (background music at 15% volume, ducked under speech)
  → visual_qa (self-review before delivery)

Smart Fallback Chain

The processor doesn't just try one method — it has a quality-aware fallback:

1. Analyze — Samples 5 frames, runs histogram analysis to detect green/blue screen
2. Try chromakey — Fast FFmpeg-based keying, checks quality score
3. Fall back to rembg — If chromakey quality < 80%, switches to AI segmentation (U2Net human seg model)
4. Alpha composite — Numpy color-distance thresholding for clean edges

This means it handles real-world footage — uneven lighting, wrinkled screens, green spill on skin — not just perfect studio setups.

---

AnimatedBackground — No More Flat Colors

The pipeline uses a new AnimatedBackground Remotion component instead of flat solid colors:

• Animated gradient mesh — Multiple radial gradients with sin/cos animated positions
• Floating glow orbs — 5 orbs with different colors, sizes, and drift speeds
• Subtle grid overlay — Pulsing opacity for depth
• Top gradient fade — Creates visual depth

The base color #0F172A auto-converts to transparent so the animated background shows through all scene types.

---

Speech-Aligned Data Visualizations

The pipeline maps your transcript timestamps to Remotion scenes, so charts and stats appear in sync with what you're saying:

• 18 scene types: hero_title, bar_chart, stat_card, callout, line_chart, progress_bar, comparison, stat_reveal, kpi_grid
• Each scene is timed to word-level ASR timestamps
• Component constraints are enforced (e.g., comparison needs 900px+ width, kpi_grid requires numeric values)

---

Try It — Sample Prompt

Record yourself on a green screen and use this prompt:

I have green screen footage at projects/my-video/assets/raw.mp4.
Here's the transcript/script of what I'm saying: [paste script]

Create an explainer video:
- Remove the green screen background
- Create animated data visualizations that match my talking points
- Composite me over the animated backgrounds (news anchor layout)
- Add TikTok-style word-by-word captions
- Mix in subtle background music
- Export at 720x1280 for Instagram Reels

---

What you need

• Zero API keys for keying — rembg runs locally with U2Net model
• FFmpeg installed
• Node.js (for Remotion background renders and caption burns)
• Python packages: rembg, numpy, Pillow (for AI segmentation fallback)

---

Updated Skill Documents

Three pipeline skill documents were updated to support green screen workflows:

• scene-director.md — New Step 0: Footage Analysis (detect green screen, speaker position, safe zones) + component constraints table
• compose-director.md — New Step 3c: Green Screen Composite pipeline + pre-flight checks (silence detection, ASR confidence, auto-corrections)
• asset-director.md — AnimatedBackground documentation + component constraints for portrait rendering

The agent now handles green screen footage automatically — no manual intervention needed.

---

Share your green screen creations in Show and tell! We'd love to see what you build.