Add multi-backend capture with automatic fallback chain

[Copilot]

Summary

Replaces hard failure on DRM initialization with graceful degradation through X11 SHM and dummy pattern backends. Enables operation in VMs, CI environments, and systems with broken/missing DRM drivers.

Details

• New feature
• Bug fix (pre-existing build issues)

What changed?

Capture Backend Abstraction

• Added capture_backend_t struct with init/capture/cleanup function pointers
• Backends try in sequence: DRM/KMS → X11 SHM → Dummy Pattern
• First successful backend is used; failure only if all backends fail

New Backends

• src/x11_capture.c: XGetImage-based screen capture for X11 systems
• src/dummy_capture.c: Animated test pattern (gradient + color bars + checkerboard)

Integration

• Moved capture/encoder initialization from main.c to service_run_host() after backend selection
• Display detection made non-fatal; fallback backends initialize display info themselves
• Backend array declared static to persist beyond function scope

Build Fixes (pre-existing)

• audio_playback.c: Renamed local typedef to avoid conflict with header
• network.c: Added missing goto in #else branch
• crypto.c: Check fgets() return value
• Makefile: Added missing src/core.c

Rationale

Current implementation exits immediately if DRM fails, making the application unusable on:

• VMs without DRM support
• Systems with broken drivers
• Users without render group permissions
• Wayland-only compositors

Fallback backends enable development/testing without display hardware and provide recovery path for production environments.

Testing

• Built successfully (make)
• Tested on:
  • Distro: Ubuntu 24.04
  • Kernel: Linux 6.8.0
  • Environment: Headless (no GPU/DRM)

Fallback chain verified:

INFO: Attempting capture backend: DRM/KMS
WARNING: Capture backend 'DRM/KMS' failed, trying next...
INFO: Attempting capture backend: X11 SHM
WARNING: Capture backend 'X11 SHM' failed, trying next...
INFO: Attempting capture backend: Dummy Pattern
✓ Dummy test pattern initialized: 1920x1080 @ 60 Hz
✓ Capture backend 'Dummy Pattern' initialized successfully

• CodeQL security scan: 0 alerts

Notes

• Potential impact: None on normal DRM path; fallback backends add ~10KB binary size
• Follow-up: X11 backend could use XShmGetImage for better performance (currently uses XGetImage)

Original prompt

PHASE 1: Multi-Fallback Display Capture Infrastructure

Current State

• ✅ drm_capture.c exists and implements DRM/KMS framebuffer capture
• ✅ service_run_host() has main streaming loop
• ❌ Missing: Fallback capture backends for when DRM fails

Problem

Currently, if DRM initialization fails, service_run_host() immediately returns -1 and exits. On systems with:

• Broken DRM drivers
• Virtual machines without DRM support
• Users without render group permissions
• Wayland-only compositors

The entire application fails with no recovery path.

Solution: Multi-Backend Capture with Automatic Fallback

1. Create capture backend abstraction struct

typedef struct {
    const char *name;
    int (*init_fn)(rootstream_ctx_t *ctx);
    int (*capture_fn)(rootstream_ctx_t *ctx, frame_buffer_t *frame);
    void (*cleanup_fn)(rootstream_ctx_t *ctx);
} capture_backend_t;

2. Implement three capture backends (in priority order):

PRIMARY: DRM/KMS (src/drm_capture.c)

• Already exists, leave as-is
• Requires root or render group membership
• Best performance/quality

FALLBACK 1: X11 SHM Screen Grab (src/x11_capture.c - NEW)

• Use XCB/Xlib to grab screen via XGetImage or XShm
• Works on X11 systems without DRM
• Slower than DRM but more compatible
• Returns RGB24/RGBA32 framebuffer

FALLBACK 2: Dummy Test Pattern (src/dummy_capture.c - NEW)

• Generates test image (gradient, moving pattern, or screenshot)
• Allows development/testing without display hardware
• Perfect for CI/headless systems
• Enables full pipeline validation

3. Implement fallback selection logic in service_run_host():

Replace hard failure:

if (rootstream_capture_init(ctx) < 0) {
    fprintf(stderr, "ERROR: Capture init failed\n");
    return -1;  // FAILS HERE
}

With resilient fallback chain:

const capture_backend_t backends[] = {
    {
        .name = "DRM/KMS",
        .init_fn = rootstream_capture_init_drm,
        .capture_fn = rootstream_capture_frame_drm,
        .cleanup_fn = rootstream_capture_cleanup_drm,
    },
    {
        .name = "X11 SHM",
        .init_fn = rootstream_capture_init_x11,
        .capture_fn = rootstream_capture_frame_x11,
        .cleanup_fn = rootstream_capture_cleanup_x11,
    },
    {
        .name = "Dummy Pattern",
        .init_fn = rootstream_capture_init_dummy,
        .capture_fn = rootstream_capture_frame_dummy,
        .cleanup_fn = rootstream_capture_cleanup_dummy,
    },
    {NULL}  // Sentinel
};

int backend_idx = 0;
while (backends[backend_idx].name) {
    printf("INFO: Attempting capture backend: %s\n", backends[backend_idx].name);
    if (backends[backend_idx].init_fn(ctx) == 0) {
        printf("✓ Capture backend '%s' initialized successfully\n", backends[backend_idx].name);
        ctx->capture_backend = &backends[backend_idx];
        break;
    } else {
        printf("WARNING: Capture backend '%s' failed, trying next...\n", backends[backend_idx].name);
        backend_idx++;
    }
}

if (!ctx->capture_backend) {
    fprintf(stderr, "ERROR: All capture backends failed!\n");
    return -1;
}

4. Update the main capture frame call to use backend:

// In streaming loop:
if (ctx->capture_backend->capture_fn(ctx, &frame) < 0) {
    fprintf(stderr, "ERROR: Frame capture failed\n");
    continue;  // Skip this frame, don't crash
}

5. Store backend selection in context:

Add to rootstream_ctx_t:

const capture_backend_t *capture_backend;  // Currently active backend

Files to Create/Modify

NEW FILES:

1. src/x11_capture.c - X11 SHM screen grab fallback
2. src/dummy_capture.c - Test pattern generator

MODIFY:

1. src/service.c - Add fallback selection logic in service_run_host()
2. include/rootstream.h - Add backend struct and function signatures
3. CMakeLists.txt - Add X11 dependency check (optional, fallback if missing)

Implementation Details

src/x11_capture.c - X11 SHM Implementation

• Open X display via XOpenDisplay()
• Get root window: DefaultRootWindow()
• Create XImage with XGetImage(display, root_window, 0, 0, width, height, ...)
• Copy pixel data to frame buffer
• Handle X11 disconnection gracefully
• Return RGBA32 format

src/dummy_capture.c - Test Pattern Implementation

• Generate simple test patterns (gradient, color bars, checkerboard)
• Optional: render text overlay (frame number, timestamp)
• Return RGBA32 format
• Useful for:
  • CI/CD testing without display
  • Development/debugging
  • Validating encoder/network without real game

Testing

1. Test DRM path: Normal operation on native Linux
2. Test X11 fallback:
   • Run in VM without DRM support
   • Verify X11 capture works
3. Test dummy pattern:
   • Run with --capture-backend=dummy
   • Verify full pipeline works without display

Success Criteria

• ✅ service_run_host() succeeds even if DRM unavailable
• ✅ At startup, prints which capture backend is ac...

This pull request was created from Copilot chat.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.