Skip to content

fix(compositor): improve image overlay quality, caching, aspect ratio, and selectability #78

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
streamer45 merged 13 commits into from
Mar 3, 2026
Merged

fix(compositor): improve image overlay quality, caching, aspect ratio, and selectability #78

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
15c80fe
fix(compositor): improve image overlay quality, caching, aspect ratio…
streamkit-devin Mar 2, 2026
ac36a3d
fix(compositor): guard against index mismatch in image overlay cache
streamkit-devin Mar 2, 2026
7915457
fix(compositor): address review — proper index mapping for cache, bro…
streamkit-devin Mar 2, 2026
c3a5d63
style(compositor): apply cargo fmt formatting
streamkit-devin Mar 2, 2026
59faac5
fix(compositor): fix HashMap type and double-deref in overlay cache
streamkit-devin Mar 2, 2026
4effd0d
fix(compositor): content-keyed overlay cache with dimension-based mat…
streamkit-devin Mar 2, 2026
6825ac2
fix(compositor): default image overlay z-index to 200 so it renders a…
streamkit-devin Mar 2, 2026
3922a50
style(compositor): add rationale comment for clippy::expect_used supp…
streamkit-devin Mar 2, 2026
7094175
style: apply formatting fixes (cargo fmt + prettier)
streamkit-devin Mar 2, 2026
e46fa89
fix(compositor): address review findings #1-#4, #7
streamkit-devin Mar 2, 2026
8eb2012
fix(compositor): preserve image aspect ratio, add image layer control…
streamkit-devin Mar 3, 2026
19e5cd0
fix(compositor): apply rotation transform to image overlay layer in c…
streamkit-devin Mar 3, 2026
291ca5a
fix(compositor): include rotationDegrees and zIndex in overlay sync c…
streamkit-devin Mar 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 84 additions & 5 deletions crates/nodes/src/video/compositor/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ use config::CompositorConfig;
use kernel::{CompositeResult, CompositeWorkItem, LayerSnapshot};
use overlay::{decode_image_overlay, rasterize_text_overlay, DecodedOverlay};
use schemars::schema_for;
use std::collections::HashMap;
use std::sync::Arc;
use streamkit_core::control::NodeControlMessage;
use streamkit_core::pins::PinManagementMessage;
Expand Down Expand Up @@ -252,8 +253,16 @@ impl ProcessorNode for CompositorNode {

// Decode image overlays (once). Wrap in Arc so per-frame clones
// into the work item are cheap reference-count bumps.
//
// `image_overlay_cfg_indices` records, for each successfully decoded
// overlay, the index of the originating `ImageOverlayConfig` in
// `config.image_overlays`. This allows the cache in
// `apply_update_params` to map decoded bitmaps back to their configs
// without relying on dimension-matching heuristics.
let mut image_overlays_vec: Vec<Arc<DecodedOverlay>> =
Vec::with_capacity(self.config.image_overlays.len());
let mut image_overlay_cfg_indices: Vec<usize> =
Vec::with_capacity(self.config.image_overlays.len());
for (i, img_cfg) in self.config.image_overlays.iter().enumerate() {
match decode_image_overlay(img_cfg) {
Ok(overlay) => {
Expand All @@ -268,6 +277,7 @@ impl ProcessorNode for CompositorNode {
overlay.rect.height,
);
image_overlays_vec.push(Arc::new(overlay));
image_overlay_cfg_indices.push(i);
},
Err(e) => {
tracing::warn!("Failed to decode image overlay {}: {}", i, e);
Expand Down Expand Up @@ -408,6 +418,7 @@ impl ProcessorNode for CompositorNode {
Self::apply_update_params(
&mut self.config,
&mut image_overlays,
&mut image_overlay_cfg_indices,
&mut text_overlays,
params,
&mut stats_tracker,
Expand Down Expand Up @@ -489,6 +500,7 @@ impl ProcessorNode for CompositorNode {
Self::apply_update_params(
&mut self.config,
&mut image_overlays,
&mut image_overlay_cfg_indices,
&mut text_overlays,
params,
&mut stats_tracker,
Expand Down Expand Up @@ -529,6 +541,7 @@ impl ProcessorNode for CompositorNode {
Self::apply_update_params(
&mut self.config,
&mut image_overlays,
&mut image_overlay_cfg_indices,
&mut text_overlays,
params,
&mut stats_tracker,
Expand Down Expand Up @@ -651,6 +664,7 @@ impl CompositorNode {
fn apply_update_params(
config: &mut CompositorConfig,
image_overlays: &mut Arc<[Arc<DecodedOverlay>]>,
image_overlay_cfg_indices: &mut Vec<usize>,
text_overlays: &mut Arc<[Arc<DecodedOverlay>]>,
params: serde_json::Value,
stats_tracker: &mut NodeStatsTracker,
Expand All @@ -666,17 +680,82 @@ impl CompositorNode {
"Updating compositor config"
);

// Always re-decode image overlays (content may have changed
// even if the count is the same).
let mut new_image_overlays =
// Re-decode image overlays only when their content or
// target rect changed. When only video-layer positions
// are updated (the common case) the existing decoded
// bitmaps are reused via Arc, avoiding redundant base64
// decode + bilinear prescale work.
//
// The cache is keyed by (data_base64, width, height).
// `image_overlay_cfg_indices` provides an exact mapping
// from each decoded overlay back to its originating
// config index, eliminating any heuristic guessing
// about which decoded bitmap belongs to which config.
let old_imgs = image_overlays.clone();
let old_cfgs = &config.image_overlays;

let mut cache: HashMap<(&str, u32, u32), Vec<Arc<DecodedOverlay>>> =
HashMap::new();

// Each decoded overlay has a recorded config index in
// `image_overlay_cfg_indices`. Use this to look up
// the originating config directly — no dimension
// matching needed.
for (dec_idx, decoded) in old_imgs.iter().enumerate() {
if let Some(&cfg_idx) = image_overlay_cfg_indices.get(dec_idx) {
if let Some(old_cfg) = old_cfgs.get(cfg_idx) {
let key = (
old_cfg.data_base64.as_str(),
old_cfg.transform.rect.width,
old_cfg.transform.rect.height,
);
cache.entry(key).or_default().push(Arc::clone(decoded));
}
}
}

let mut new_image_overlays: Vec<Arc<DecodedOverlay>> =
Vec::with_capacity(new_config.image_overlays.len());
let mut new_cfg_indices: Vec<usize> =
Vec::with_capacity(new_config.image_overlays.len());
for img_cfg in &new_config.image_overlays {
for (new_idx, img_cfg) in new_config.image_overlays.iter().enumerate() {
let key = (
img_cfg.data_base64.as_str(),
img_cfg.transform.rect.width,
img_cfg.transform.rect.height,
);
if let Some(entries) = cache.get_mut(&key) {
if let Some(existing) = entries.pop() {
// Content and target dimensions unchanged —
// reuse the decoded bitmap. The overlay's
// rect may be smaller than the config rect
// due to aspect-ratio-preserving prescale,
// so re-centre within the new config rect.
let mut ov = (*existing).clone();
let cfg_w = img_cfg.transform.rect.width.cast_signed();
let cfg_h = img_cfg.transform.rect.height.cast_signed();
let ov_w = ov.rect.width.cast_signed();
let ov_h = ov.rect.height.cast_signed();
ov.rect.x = img_cfg.transform.rect.x + (cfg_w - ov_w) / 2;
ov.rect.y = img_cfg.transform.rect.y + (cfg_h - ov_h) / 2;
ov.opacity = img_cfg.transform.opacity;
ov.rotation_degrees = img_cfg.transform.rotation_degrees;
ov.z_index = img_cfg.transform.z_index;
new_image_overlays.push(Arc::new(ov));
new_cfg_indices.push(new_idx);
continue;
}
}
match decode_image_overlay(img_cfg) {
Ok(ov) => new_image_overlays.push(Arc::new(ov)),
Ok(ov) => {
new_image_overlays.push(Arc::new(ov));
new_cfg_indices.push(new_idx);
},
Err(e) => tracing::warn!("Image overlay decode failed: {e}"),
}
}
*image_overlays = Arc::from(new_image_overlays);
*image_overlay_cfg_indices = new_cfg_indices;

// Re-rasterize text overlays.
let new_text_overlays: Vec<Arc<DecodedOverlay>> = new_config
Expand Down
76 changes: 51 additions & 25 deletions crates/nodes/src/video/compositor/overlay.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,18 +47,49 @@ pub fn decode_image_overlay(config: &ImageOverlayConfig) -> Result<DecodedOverla
let target_w = config.transform.rect.width;
let target_h = config.transform.rect.height;

// Pre-scale the decoded image to the target rect dimensions so that
// the per-frame `scale_blit_rgba_rotated` call hits the identity-scale
// fast path (direct memcpy) instead of doing nearest-neighbor scaling
// every frame.
// Pre-scale the decoded image to fit within the target rect while
// preserving the source aspect ratio. This ensures the per-frame
// `scale_blit_rgba_rotated` call hits the identity-scale fast path
// (direct memcpy) and the image is never stretched.
if target_w > 0 && target_h > 0 && (w != target_w || h != target_h) {
// Aspect-ratio-preserving fit: scale so the image fits inside
// the target box without distortion.
#[allow(clippy::cast_precision_loss)]
let scale = {
let sw = w as f32;
let sh = h as f32;
(target_w as f32 / sw).min(target_h as f32 / sh)
};
#[allow(
clippy::cast_possible_truncation,
clippy::cast_sign_loss,
clippy::cast_precision_loss
)]
let fit_w = ((w as f32 * scale).round() as u32).max(1);
#[allow(
clippy::cast_possible_truncation,
clippy::cast_sign_loss,
clippy::cast_precision_loss
)]
let fit_h = ((h as f32 * scale).round() as u32).max(1);

let raw = rgba.into_raw();
let scaled = prescale_rgba(&raw, w, h, target_w, target_h);
let scaled = prescale_rgba(&raw, w, h, fit_w, fit_h);

// Adjust the rect to match the fitted dimensions so the blit
// stays on the identity-scale path and the image is centred
// within the originally requested area.
let mut rect = config.transform.rect.clone();
rect.x += (target_w.cast_signed() - fit_w.cast_signed()) / 2;
rect.y += (target_h.cast_signed() - fit_h.cast_signed()) / 2;
rect.width = fit_w;
rect.height = fit_h;

Ok(DecodedOverlay {
rgba_data: scaled,
width: target_w,
height: target_h,
rect: config.transform.rect.clone(),
width: fit_w,
height: fit_h,
rect,
opacity: config.transform.opacity,
rotation_degrees: config.transform.rotation_degrees,
z_index: config.transform.z_index,
Expand All @@ -76,24 +107,19 @@ pub fn decode_image_overlay(config: &ImageOverlayConfig) -> Result<DecodedOverla
}
}

/// Nearest-neighbor scale an RGBA8 buffer from `(sw, sh)` to `(dw, dh)`.
/// Used once at config time so the per-frame blit is a 1:1 copy.
/// Bilinear-filtered scale of an RGBA8 buffer from `(sw, sh)` to `(dw, dh)`.
/// Uses the `image` crate's `resize` with `Triangle` (bilinear) filter for
/// high-quality prescaling — much better than nearest-neighbor for images
/// containing text or fine detail. Called once at config time so the
/// per-frame blit is a 1:1 copy.
fn prescale_rgba(src: &[u8], sw: u32, sh: u32, dw: u32, dh: u32) -> Vec<u8> {
let sw = sw as usize;
let sh = sh as usize;
let dw = dw as usize;
let dh = dh as usize;
let mut out = vec![0u8; dw * dh * 4];
for dy in 0..dh {
let sy = dy * sh / dh;
for dx in 0..dw {
let sx = dx * sw / dw;
let si = (sy * sw + sx) * 4;
let di = (dy * dw + dx) * 4;
out[di..di + 4].copy_from_slice(&src[si..si + 4]);
}
}
out
// Invariant: caller guarantees src.len() == sw * sh * 4.
#[allow(clippy::expect_used)]
Comment thread
staging-devin-ai-integration[bot] marked this conversation as resolved.
// from_raw only fails if buffer length != w*h*4; caller guarantees this
let src_img = image::RgbaImage::from_raw(sw, sh, src.to_vec())
.expect("prescale_rgba: source dimensions do not match buffer length");
let resized = image::imageops::resize(&src_img, dw, dh, image::imageops::FilterType::Triangle);
resized.into_raw()
}

// ── Bundled default font ────────────────────────────────────────────────────
Expand Down
96 changes: 58 additions & 38 deletions ui/src/components/CompositorCanvas.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,18 +111,6 @@ const InlineTextInput = styled.textarea`
font-family: inherit;
`;

/** Icon badge for image overlay layers */
const ImageBadge = styled.div`
position: absolute;
inset: 0;
display: flex;
align-items: center;
justify-content: center;
pointer-events: none;
z-index: 1;
opacity: 0.5;
`;

const LayerDimensions = styled.div`
position: absolute;
bottom: 2px;
Expand Down Expand Up @@ -201,26 +189,6 @@ const ResizeHandles: React.FC<{
));
ResizeHandles.displayName = 'ResizeHandles';

// ── Image icon SVG ──────────────────────────────────────────────────────────

const ImageIcon: React.FC<{ size?: number }> = ({ size = 24 }) => (
<svg
width={size}
height={size}
viewBox="0 0 24 24"
fill="none"
stroke="currentColor"
strokeWidth="1.5"
strokeLinecap="round"
strokeLinejoin="round"
style={{ color: 'rgba(255,255,255,0.5)' }}
>
<rect x="3" y="3" width="18" height="18" rx="2" ry="2" />
<circle cx="8.5" cy="8.5" r="1.5" />
<polyline points="21 15 16 10 5 21" />
</svg>
);

// ── Video input layer ───────────────────────────────────────────────────────

const VideoLayer: React.FC<{
Expand Down Expand Up @@ -437,6 +405,46 @@ const ImageOverlayLayer: React.FC<{
const borderColor = isSelected ? 'var(--sk-primary)' : `hsla(${hue}, 70%, 65%, 0.8)`;
const bgColor = isSelected ? `hsla(${hue}, 60%, 50%, 0.25)` : `hsla(${hue}, 60%, 50%, 0.12)`;

// Build a blob URL for the image thumbnail. Using fetch() with a
// data-URI lets the browser decode the base64 natively, which is
// more efficient than the manual atob() + byte-by-byte Uint8Array
// copy for large images.
//
// MIME detection: we inspect the base64-encoded magic bytes at the
// start of the string to pick the correct MIME type. The fallback
// is JPEG, which covers the common `/9j/` prefix and variants.
const [imgSrc, setImgSrc] = useState<string | undefined>();

useEffect(() => {
if (!overlay.dataBase64) {
setImgSrc(undefined);
return;
}
let mime = 'image/jpeg'; // default fallback
if (overlay.dataBase64.startsWith('iVBOR')) mime = 'image/png';
else if (overlay.dataBase64.startsWith('R0lGOD')) mime = 'image/gif';
else if (overlay.dataBase64.startsWith('UklGR')) mime = 'image/webp';

let cancelled = false;
let url: string | undefined;

fetch(`data:${mime};base64,${overlay.dataBase64}`)
.then((r) => r.blob())
.then((blob) => {
if (cancelled) return;
url = URL.createObjectURL(blob);
setImgSrc(url);
})
.catch(() => {
// Ignore decode failures — no thumbnail shown.
});

return () => {
cancelled = true;
if (url) URL.revokeObjectURL(url);
};
}, [overlay.dataBase64]);
Copy link
Copy Markdown
Contributor Author

@staging-devin-ai-integration staging-devin-ai-integration bot Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚩 Large base64 strings passed through React state and props without memoization

The ImageOverlayState.dataBase64 field (potentially multi-MB) flows through React state (setImageOverlays), is stored in refs (imageOverlaysRef), serialized in serializeImageOverlays, compared in the sync effect, and passed as a prop to ImageOverlayLayer. The useEffect dependency at ui/src/components/CompositorCanvas.tsx:446 triggers on overlay.dataBase64 changes, requiring React to compare potentially large strings on every render. For multi-MB images this could cause performance issues. The fetch(data:...) approach amortizes the decode cost but doesn't address the comparison cost.

Staging: Open in Devin

Was this helpful? React with 👍 or 👎 to provide feedback.

Debug

Playground


return (
<LayerBox
ref={layerRef}
Expand All @@ -447,19 +455,31 @@ const ImageOverlayLayer: React.FC<{
width: overlay.width,
height: overlay.height,
opacity: overlay.visible ? overlay.opacity : 0.2,
transform:
overlay.rotationDegrees !== 0 ? `rotate(${overlay.rotationDegrees}deg)` : undefined,
zIndex: overlay.zIndex ?? 200 + index,
border: `2px dotted ${borderColor}`,
border: `2px solid ${borderColor}`,
background: bgColor,
filter: overlay.visible ? undefined : 'grayscale(0.6)',
backgroundImage:
'repeating-linear-gradient(45deg, transparent, transparent 6px, rgba(255,255,255,0.04) 6px, rgba(255,255,255,0.04) 12px)',
}}
onPointerDown={handlePointerDown}
>
{imgSrc && (
<img
src={imgSrc}
alt={`Image overlay ${index}`}
style={{
position: 'absolute',
inset: 0,
width: '100%',
height: '100%',
objectFit: 'contain',
pointerEvents: 'none',
opacity: 0.85,
}}
/>
)}
<LayerLabel>IMG #{index}</LayerLabel>
<ImageBadge>
<ImageIcon size={24} />
</ImageBadge>
{isSelected && <ResizeHandles layerId={overlay.id} onResizeStart={onResizeStart} />}
</LayerBox>
);
Expand Down
Loading