Fix compilation errors preventing multi-fallback capture build

[Copilot]

Summary

Three compilation errors blocked building the existing DRM → X11 → Dummy capture fallback system. Fixed variable redeclarations and missing function implementation.

Details

• Bug fix
• New feature
• Performance improvement
• Documentation / tooling

What changed?

1. src/vaapi_encoder.c:228 - Duplicate VAStatus status declaration

// Before: VAStatus status = vaQueryConfigProfiles(...)
// After:  status = vaQueryConfigProfiles(...)

2. src/service.c:250 - Conflicting loop variable in encoder fallback

// Renamed encoder backend loop from `backend_idx` to `encoder_idx`
// Capture loop already used `backend_idx`

3. src/audio_playback.c - Missing linker symbol

bool audio_playback_alsa_available(void) {
    snd_pcm_t *test_handle = NULL;
    int err = snd_pcm_open(&test_handle, "default", 
                           SND_PCM_STREAM_PLAYBACK, SND_PCM_NONBLOCK);
    if (err < 0) return false;
    snd_pcm_close(test_handle);
    return true;
}

4. .gitignore - Exclude CodeQL build artifacts

Rationale

The three-tier capture fallback (DRM/KMS → X11 SHM → Dummy) was fully implemented but couldn't build. Minimal surgical fixes restore buildability without touching capture logic.

Aligns with RootStream's simplicity goal - no workarounds, just fix the errors.

Testing

• Built successfully (make)
• Unit tests pass (3/3: crypto, encoding, packet)
• CodeQL security scan clean (0 vulnerabilities)
• Basic streaming tested
• Tested on:
  • Distro: Ubuntu 24.04 (build environment)
  • Kernel: N/A (build-only fixes)
  • GPU & driver: N/A (build-only fixes)

Notes

• Potential impact on latency or resource usage: None - build fixes only
• Any follow-up work needed: Runtime testing of fallback chain on systems without DRM (VMs, headless)

Original prompt

PHASE 1: Multi-Fallback Display Capture Implementation

Current State

The project has a working DRM/KMS capture implementation in src/drm_capture.c, but if DRM fails (VM, broken drivers, no permissions), the entire application exits with no recovery path.

Problem

In service_run_host():

if (rootstream_capture_init(ctx) < 0) {
    fprintf(stderr, "ERROR: Capture init failed\n");
    return -1;  // HARD FAIL - application exits
}

Systems affected:

• Virtual machines without DRM support
• Users without render group membership
• Broken/missing GPU drivers
• Headless servers (CI/CD, SSH)
• Wayland-only systems without DRM

Solution: Three-Tier Capture Fallback

Tier 1: DRM/KMS (Primary)

• File: src/drm_capture.c (already exists)
• Status: ✅ Complete
• Performance: Best (direct kernel access)
• Availability: High-end Linux systems with modern drivers

Tier 2: X11 SHM Fallback (New)

• File: src/x11_capture.c (NEW)
• Method: XGetImage or X11 SHM
• Performance: Medium (slower than DRM)
• Availability: Any X11 system

Tier 3: Dummy Test Pattern (New)

• File: src/dummy_capture.c (NEW)
• Method: Generate test pattern (checkerboard + timestamp)
• Performance: Fast (no I/O)
• Availability: Always (no dependencies)

Implementation Details

File 1: src/x11_capture.c - X11 SHM Screen Grab

/*
 * x11_capture.c - X11 SHM screen capture fallback
 * 
 * Uses XGetImage or XShmGetImage to grab screen.
 * Falls back to XGetImage if XShm unavailable.
 * No compositor dependency - works on X11 directly.
 */

#include "../include/rootstream.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

#ifdef HAVE_X11
#include <X11/Xlib.h>
#include <X11/Xutil.h>
#ifdef HAVE_X11_SHM
#include <X11/extensions/XShm.h>
#include <sys/ipc.h>
#include <sys/shm.h>
#endif
#endif

typedef struct {
    Display *display;
    Window root;
    int width;
    int height;
    int depth;
    XImage *image;
#ifdef HAVE_X11_SHM
    XShmSegmentInfo shminfo;
    bool use_shm;
#endif
} x11_capture_ctx_t;

static void set_error(const char *msg) {
    /* Store error for rootstream_get_error() */
}

/*
 * Initialize X11 capture
 */
int rootstream_capture_init_x11(rootstream_ctx_t *ctx) {
#ifndef HAVE_X11
    set_error("X11 support not compiled in");
    return -1;
#else
    x11_capture_ctx_t *x11 = calloc(1, sizeof(x11_capture_ctx_t));
    if (!x11) {
        set_error("Memory allocation failed");
        return -1;
    }

    /* Open X display */
    x11->display = XOpenDisplay(NULL);
    if (!x11->display) {
        set_error("Cannot open X11 display");
        free(x11);
        return -1;
    }

    /* Get root window */
    int screen = DefaultScreen(x11->display);
    x11->root = RootWindow(x11->display, screen);
    
    /* Get window attributes */
    XWindowAttributes attrs;
    if (!XGetWindowAttributes(x11->display, x11->root, &attrs)) {
        set_error("Cannot get root window attributes");
        XCloseDisplay(x11->display);
        free(x11);
        return -1;
    }

    x11->width = attrs.width;
    x11->height = attrs.height;
    x11->depth = attrs.depth;

    /* Try XShm */
#ifdef HAVE_X11_SHM
    if (XShmQueryExtension(x11->display)) {
        x11->image = XShmCreateImage(x11->display, attrs.visual,
                                     x11->depth, ZPixmap, NULL,
                                     &x11->shminfo, x11->width, x11->height);
        if (x11->image) {
            x11->shminfo.shmid = shmget(IPC_PRIVATE,
                                       x11->image->bytes_per_line * x11->height,
                                       IPC_CREAT | 0777);
            if (x11->shminfo.shmid >= 0) {
                x11->shminfo.shmaddr = x11->image->data = 
                    (char *)shmat(x11->shminfo.shmid, 0, 0);
                
                if (x11->shminfo.shmaddr != (char *)-1) {
                    x11->shminfo.readOnly = False;
                    if (XShmAttach(x11->display, &x11->shminfo)) {
                        x11->use_shm = true;
                        printf("✓ X11 capture using XShm\n");
                    }
                }
            }
        }
    }
#endif

    /* Fallback to regular XGetImage if XShm unavailable/failed */
    if (!x11->image) {
#ifdef HAVE_X11_SHM
        x11->use_shm = false;
#endif
        x11->image = XGetImage(x11->display, x11->root, 0, 0,
                              x11->width, x11->height, AllPlanes, ZPixmap);
        if (!x11->image) {
            set_error("XGetImage failed");
            XCloseDisplay(x11->display);
            free(x11);
            return -1;
        }
        printf("✓ X11 capture using XGetImage\n");
    }

    /* Allocate frame buffer */
    size_t frame_size = x11->width * x11->height * 4;  /* RGBA */
    ctx->current_frame.data = malloc(frame_size);
    if (!ctx->current_frame.data) {
        set_error("Frame buffer allocation failed"...

</details>



<!-- START COPILOT CODING AGENT SUFFIX -->

*This pull request was created from Copilot chat.*
>

<!-- START COPILOT CODING AGENT TIPS -->
---

✨ Let Copilot coding agent [set things up for you](https://github.com/infinityabundance/RootStream/issues/new?title=✨+Set+up+Copilot+instructions&body=Configure%20instructions%20for%20this%20repository%20as%20documented%20in%20%5BBest%20practices%20for%20Copilot%20coding%20agent%20in%20your%20repository%5D%28https://gh.io/copilot-coding-agent-tips%29%2E%0A%0A%3COnboard%20this%20repo%3E&assignees=copilot) — coding agent works faster and does higher quality work when set up for your repo.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.