v2 roadmap

[endernoke]

Overview

The current architecture of ink-picture struggles with the way terminal bitmap protocols interact with Ink's rendering loop. Here are my thoughts on a V2 rewrite to sustainably fix the bitmap related issues and various DX problems. The final goal is to make images a non-hacky and first-class citizen in ink.

Current Problems

UX

• Image Flickering: Protocols like Sixel, iTerm2, and Kitty bypass React’s rendering cycle
  • When Ink clears the terminal per-update, the image vanishes until our setTimeout hack re-renders it. This is unstable and visually jarring
• Cursor Movement: We walk up the entire component tree to calculate the current absolute position in the terminal and move the cursor to it to blit image escape sequences at the right location. This implement may have performance implications and is not well-implemented (see Dislocated images when using high-res protocols supreme-gg-gg/instagram-cli#158)
• Layout Fragility: Current components often collapse inside containers, and lack relative sizing support

DX

• Provider Antipattern: TerminalInfoProvider throws an error after a timeout if terminal info queried via escape sequences are not received (which could be flaky at times)
• Dependencies: sharp provides great performance via native binaries but faces cross-platform installation issues (see sharp installation issue #11 Sharp Compatibility #16 refactor: replace sharp with jimp #17)

Proposed Solutions

Architecture

• Add/contribute an OpaqueSurface or similar component to upstream ink, which:
  • Tells Ink to ignore a specific coordinate range during its clear/render cycle
  • Exposes APIs for efficiently manipulating the content inside this region
    This allows us to draw bitmaps without interference.
• Move away from the mandatory Provider. Use a progressive enhancement strategy with safe default fallbacks, asynchronous updates, able to being overridden with explicit environment variables and/or passing down values from downstream applications

Image Processing Dependency

• Evaluate replacing sharp with a more portable solution:
  • jimp (Pure JS, easy install, slower)
  • Or make the image processor pluggable so users can choose

Code Quality

• De-slop protocol selection logic
• Rewrite components to better utilize Ink's Yoga layout engine for consistent sizing

Tentative Milestones

1. Prototype the OpaqueSurface in ink and validate if it solves the flickering
2. Propose and merge a PR to Ink (will be blocking)
3. Rewrite image rendering using the new Ink APIs
4. Implement the new TerminalInfoProvider or equivalent
5. Implement the Image component using the new code, with better layout detection

This proposal is in no way final. Would love to hear any thoughts, suggestions, or alternative designs. Also feel free to comment here if you'd like to get involved.