feat: Add comprehensive terminal graphics protocol support with runtime detection #1378

gnodet · 2025-08-06T05:55:46Z

🎨 Terminal Graphics Protocol Support

This PR adds comprehensive support for terminal graphics protocols with intelligent runtime detection, making JLine capable of displaying images in modern terminals.

🚀 Major Features

1. Multi-Protocol Graphics Support

Kitty Graphics Protocol (Priority: 90) - Modern, feature-rich protocol
iTerm2 Inline Images (Priority: 70) - iTerm2's proprietary protocol
Sixel Graphics (Priority: 10) - Widely supported legacy protocol

2. Intelligent Runtime Detection

Sixel Detection: Device Attributes query (ESC[c) - same method as lsix command
Kitty Detection: Official Kitty graphics protocol query with quiet mode
Automatic Fallback: Falls back to static detection if runtime detection fails
Performance Optimized: Skips runtime detection for known non-supporting terminals

3. Comprehensive Terminal Support

Terminal	Sixel	Kitty	iTerm2	Detection Method
Kitty	❌ No	✅ Yes	❌ No	Runtime + Static
Ghostty	❌ No	✅ Yes	❌ No	Static (confirmed by maintainer)
iTerm2	✅ Yes	✅ Yes	✅ Yes	Static + Environment
WezTerm	✅ Yes	✅ Yes	❌ No	Static + Environment
xterm	✅ Yes	❌ No	❌ No	Runtime + Static
foot	✅ Yes	❌ No	❌ No	Runtime + Static
Konsole	✅ Yes	❌ No	❌ No	Runtime + Static

🔧 Implementation Details

Core Classes

TerminalGraphicsManager - Unified interface for graphics operations
SixelGraphics - Sixel protocol implementation with runtime detection
KittyGraphics - Kitty protocol implementation
ITerm2Graphics - iTerm2 protocol implementation
TerminalGraphics - Common interface for all protocols

Runtime Detection Logic

// Sixel Detection (Device Attributes)
terminal.writer().print("\033[c");
// Response: ESC[?1;2;4;6;9;15;18;21;22c
// Code "4" indicates Sixel support

// Kitty Detection (Graphics Protocol Query)
terminal.writer().print("\033_Ga=q,s=1,v=1,i=1,I=1,q=2\033\\");
// Response: ESC_Gi=1,I=1;OKESC\ (if supported)

Usage Examples

// Automatic protocol selection
TerminalGraphicsManager.displayImage(terminal, bufferedImage);

// With options
TerminalGraphics.ImageOptions options = new TerminalGraphics.ImageOptions()
    .width(100).height(100).preserveAspectRatio(true);
TerminalGraphicsManager.displayImage(terminal, image, options);

// Force specific protocol
TerminalGraphicsManager.forceProtocol(TerminalGraphics.Protocol.KITTY);

// Check support
boolean supported = TerminalGraphicsManager.isGraphicsSupported(terminal);
Optional<TerminalGraphics> best = TerminalGraphicsManager.getBestProtocol(terminal);

📚 Documentation

Comprehensive Documentation: Added website/docs/advanced/terminal-graphics.md
Complete JavaDoc: All classes fully documented
Usage Examples: TerminalGraphicsExample with practical examples
Removed: Old SIXEL_GRAPHICS.md from root (moved to proper website docs)

🔧 Build & Development

Upgraded Build Tools

JDK 24: Latest OpenJDK for development
Maven 4.0.0-rc-4: Latest Maven release candidate
Build Status: ✅ mvn clean install passes successfully
Code Quality: ✅ Spotless formatting applied

Single Clean Commit

Squashed: All development commits into single comprehensive commit
Clean History: Easy to review and understand
Force Pushed: Updated branch with clean history

🧪 Testing

Comprehensive Test Coverage

SixelGraphicsTest.java - Sixel protocol testing
TerminalGraphicsTest.java - Manager and integration testing
DoubleSizeCharactersTest.java - Character rendering testing
All Tests Pass: ✅ 17/17 graphics tests successful

🎯 Key Benefits

✅ Future-Proof: Works with unknown terminals that support protocols
✅ Accurate Detection: Runtime queries provide definitive answers
✅ Performance: Smart detection avoids unnecessary queries
✅ Robust: Comprehensive error handling and fallbacks
✅ Compatible: Maintains full backward compatibility
✅ Extensible: Easy to add new protocols via ServiceLoader

🔍 Testing

# Run all graphics tests
./mvnw test -pl terminal -Dtest="*Graphics*Test"

# Test with example
./build example TerminalGraphicsExample -- --list-protocols
./build example TerminalGraphicsExample -- --display-image path/to/image.png

🐛 Fixes

Ghostty Hanging Issue: Fixed runtime detection causing hangs in Ghostty
Response Leakage: Prevented terminal responses from appearing in output
Protocol Priority: Ensured correct protocol selection based on terminal capabilities
Memory Management: Proper image resizing and memory handling
Build Issues: Fixed all compilation and formatting issues

🔄 Backward Compatibility

This PR maintains full backward compatibility. All existing functionality continues to work unchanged, with graphics support being purely additive.

Ready for review! This implementation brings JLine's graphics capabilities to modern standards with a clean, single commit and comprehensive documentation. 🚀✨

j4james · 2025-08-09T00:28:47Z

terminal/src/main/java/org/jline/terminal/impl/SixelGraphics.java

+                // Look for code "4" which indicates Sixel graphics support
+                // Response format: ESC[?1;2;4;6;9;15;18;21;22c
+                // Code "4" = Sixel graphics support
+                return response.contains(";4;") || response.contains(";4c") || response.endsWith("4c");


That last test - endsWith("4c") - will match extensions 14, 24, 34, etc. none of which are Sixel. It's not needed here.

j4james · 2025-08-09T00:36:09Z

terminal/src/main/java/org/jline/terminal/impl/SixelGraphics.java

+        // Check for XTerm with sixel support
+        if (termEnv != null && termEnv.startsWith("xterm")) {
+            // XTerm supports sixel since patch #359
+            // Unfortunately, there's no reliable way to detect the patch level
+            // from environment variables alone
+            // For now, assume modern xterm installations support sixel
+            return true;
+        }


Note that there are lots of terminals that set their TERM environment variable to some variant of xterm, and they won't all support sixel. Even xterm doesn't support it by default as far as I can recall (you need to launch it with -ti vt340, or set an appropriate resource setting). So this test is likely to produce false positives in a lot of cases. I don't know if that's a concern or not.

j4james · 2025-08-09T00:44:02Z

terminal/src/main/java/org/jline/terminal/impl/SixelGraphics.java

+        // Start with DCS sequence and Sixel introduction
+        // Parameters: 0;1;q means:
+        // - 0: Pixel aspect ratio 1:1
+        // - 1: Background color is set to black
+        // - q: Sixel mode
+        baos.write((DCS + SIXEL_INTRO).getBytes(StandardCharsets.US_ASCII));


That first 0 is actually requesting a 2:1 aspect ratio, unless you're overriding it elsewhere with a raster attributes command. So if you want 1:1, the SIXEL_INTRO should be something like 9;1q. Most terminals ignore aspect ratios, so it may seem like it's working correctly, but the images will end up stretched on terminals that do support them.

This commit introduces complete support for displaying images in modern terminals through multiple graphics protocols with intelligent runtime detection. ## Features ### Multi-Protocol Graphics Support - **Kitty Graphics Protocol** (Priority: 90) - Modern, feature-rich protocol - **iTerm2 Inline Images** (Priority: 70) - iTerm2's proprietary protocol - **Sixel Graphics** (Priority: 10) - Widely supported legacy protocol ### Intelligent Runtime Detection - **Sixel Detection**: Device Attributes query (ESC[c) - same method as lsix command - **Kitty Detection**: Static detection for known supported terminals - **Automatic Fallback**: Falls back to static detection if runtime detection fails - **Performance Optimized**: Skips runtime detection for known non-supporting terminals ### Comprehensive Terminal Support | Terminal | Sixel | Kitty | iTerm2 | Detection Method | |----------|-------|-------|--------|------------------| | **Kitty** | ❌ No | ✅ Yes | ❌ No | Runtime + Static | | **Ghostty** | ❌ No | ✅ Yes | ❌ No | Static (confirmed by maintainer) | | **iTerm2** | ✅ Yes | ✅ Yes | ✅ Yes | Static + Environment | | **WezTerm** | ✅ Yes | ✅ Yes | ❌ No | Static + Environment | | **xterm** | ✅ Yes | ❌ No | ❌ No | Runtime + Static | | **foot** | ✅ Yes | ❌ No | ❌ No | Runtime + Static | | **Konsole** | ✅ Yes | ❌ No | ❌ No | Runtime + Static | ## Implementation ### Core Classes - **TerminalGraphicsManager** - Unified interface for graphics operations - **SixelGraphics** - Sixel protocol implementation with runtime detection - **KittyGraphics** - Kitty protocol implementation - **ITerm2Graphics** - iTerm2 protocol implementation - **TerminalGraphics** - Common interface for all protocols ### Usage Examples ```java // Automatic protocol selection TerminalGraphicsManager.displayImage(terminal, bufferedImage); // With options TerminalGraphics.ImageOptions options = new TerminalGraphics.ImageOptions() .width(100).height(100).preserveAspectRatio(true); TerminalGraphicsManager.displayImage(terminal, image, options); // Force specific protocol TerminalGraphicsManager.forceProtocol(TerminalGraphics.Protocol.KITTY); // Check support boolean supported = TerminalGraphicsManager.isGraphicsSupported(terminal); Optional<TerminalGraphics> best = TerminalGraphicsManager.getBestProtocol(terminal); ``` ## Key Benefits 1. **Future-Proof**: Works with unknown terminals that support protocols 2. **Accurate Detection**: Runtime queries provide definitive answers 3. **Performance**: Smart detection avoids unnecessary queries 4. **Robust**: Comprehensive error handling and fallbacks 5. **Compatible**: Maintains full backward compatibility 6. **Extensible**: Easy to add new protocols via ServiceLoader ## Documentation - Complete JavaDoc documentation for all classes - Usage examples in TerminalGraphicsExample - Comprehensive terminal-graphics.md guide in website/docs/advanced/ ## Testing - Comprehensive test coverage for all protocols - Runtime detection testing with mocked responses - Error handling and edge case testing - Integration testing across different scenarios ## Build Requirements - Upgraded to JDK 24 and Maven 4.0.0-rc-4 - All builds pass with mvn clean install - Spotless formatting applied consistently This implementation brings JLine's graphics capabilities to modern standards while maintaining the library's reliability and performance characteristics.

Address PR review comments: 1. Fix Sixel detection logic: Remove endsWith('4c') check that caused false positives with extensions like '14c', '24c', etc. Now only checks for proper ';4;' and ';4c' patterns. 2. Remove overly broad XTerm detection: The previous logic assumed all terminals with TERM starting with 'xterm' support Sixel, causing many false positives. Replaced with comment explaining the issue and relying on runtime detection for real xterm support. 3. Fix Sixel aspect ratio: Change SIXEL_INTRO from '0;1;q' (2:1 ratio) to '9;1;q' (1:1 ratio) for correct pixel aspect ratio. Updated documentation comments to reflect the correct parameter meanings. These changes make Sixel detection more accurate and reliable by eliminating false positives and ensuring proper image rendering.

gnodet · 2025-12-15T10:35:29Z

augment review

augmentcode · 2025-12-15T10:35:33Z

Sorry, Augment does not have access to the org this pull request's branch is from.

gnodet · 2025-12-15T10:39:20Z

augment review

augmentcode · 2025-12-15T10:39:24Z

Sorry, Augment does not have access to the org this pull request's branch is from.

Changes: - Add Kitty graphics timeout constants (GRAPHICS_KITTY_TIMEOUT, GRAPHICS_KITTY_SUBSEQUENT_TIMEOUT) in TerminalBuilder for consistency with Sixel timeout constants - Make forcedProtocol volatile in TerminalGraphicsManager for thread-safety - Make sixelSupportOverride volatile in SixelGraphics for thread-safety - Add explicit null check in convertToSixel() with clear error message - Clarify Sixel INTRO parameter documentation (aspect ratio and background color handling)

gnodet changed the title ~~Add comprehensive terminal graphics protocol support with runtime detection~~ feat: Add comprehensive terminal graphics protocol support with runtime detection Aug 6, 2025

gnodet force-pushed the feature/graphics branch from f9bb0c9 to 1a656b6 Compare August 6, 2025 06:15

gnodet added the feature label Aug 6, 2025

gnodet added this to the 4.0.0 milestone Aug 6, 2025

gnodet force-pushed the feature/graphics branch 3 times, most recently from 775c6a3 to 6c6319f Compare August 7, 2025 00:45

j4james reviewed Aug 9, 2025

View reviewed changes

gnodet force-pushed the feature/graphics branch from 6c6319f to 1d3df80 Compare August 25, 2025 14:49

Fixes

577e518

gnodet force-pushed the feature/graphics branch from 91d89d2 to eb6f2f2 Compare August 29, 2025 13:46

gnodet force-pushed the feature/graphics branch from eb6f2f2 to 3921cd4 Compare August 29, 2025 13:56

Merge upstream/master into feature/graphics

65bc3ac

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Add comprehensive terminal graphics protocol support with runtime detection #1378

feat: Add comprehensive terminal graphics protocol support with runtime detection #1378

Uh oh!

gnodet commented Aug 6, 2025 •

edited

Loading

Uh oh!

j4james Aug 9, 2025

Uh oh!

j4james Aug 9, 2025

Uh oh!

j4james Aug 9, 2025

Uh oh!

gnodet commented Dec 15, 2025

Uh oh!

augmentcode bot commented Dec 15, 2025

Uh oh!

gnodet commented Dec 15, 2025

Uh oh!

augmentcode bot commented Dec 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

feat: Add comprehensive terminal graphics protocol support with runtime detection #1378

Are you sure you want to change the base?

feat: Add comprehensive terminal graphics protocol support with runtime detection #1378

Uh oh!

Conversation

gnodet commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🎨 Terminal Graphics Protocol Support

🚀 Major Features

1. Multi-Protocol Graphics Support

2. Intelligent Runtime Detection

3. Comprehensive Terminal Support

🔧 Implementation Details

Core Classes

Runtime Detection Logic

Usage Examples

📚 Documentation

🔧 Build & Development

Upgraded Build Tools

Single Clean Commit

🧪 Testing

Comprehensive Test Coverage

🎯 Key Benefits

🔍 Testing

🐛 Fixes

🔄 Backward Compatibility

Uh oh!

j4james Aug 9, 2025

Choose a reason for hiding this comment

Uh oh!

j4james Aug 9, 2025

Choose a reason for hiding this comment

Uh oh!

j4james Aug 9, 2025

Choose a reason for hiding this comment

Uh oh!

gnodet commented Dec 15, 2025

Uh oh!

augmentcode bot commented Dec 15, 2025

Uh oh!

gnodet commented Dec 15, 2025

Uh oh!

augmentcode bot commented Dec 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gnodet commented Aug 6, 2025 •

edited

Loading