Skip to content

Make wgpu GPU-rendered frames visible via transparent webview overlay #14

@CREVIOS

Description

@CREVIOS

Problem

The wgpu GPU renderer is fully wired and running — it creates a Metal surface on the Tauri window, uploads frame data via write_texture(), and renders a fullscreen quad. But the frames are invisible because the webview is opaque and covers the entire window.

When we disabled the IPC frame path (to avoid duplicate work), the screen went black because the webview was rendering nothing and blocking the wgpu surface underneath.

Goal

Make the wgpu-rendered frames visible by making the webview transparent in the session area, so the GPU-rendered content shows through.

Architecture

┌──── Window ─────────────────────────────┐
│                                          │
│  LAYER 0: wgpu Metal Surface             │
│  ┌──────────────────────────────────┐    │
│  │  GPU-rendered RDP frames          │    │
│  │  (fullscreen textured quad)       │    │
│  └──────────────────────────────────┘    │
│                                          │
│  LAYER 1: WebView (transparent bg)       │
│  ┌──────────────────────────────────┐    │
│  │  ┌─────┐ Toolbar (opaque)         │    │
│  │  │ Side│ ┌────────────────────┐   │    │
│  │  │ bar │ │                    │   │    │
│  │  │     │ │  TRANSPARENT AREA  │   │    │
│  │  │     │ │  (GPU shows thru)  │   │    │
│  │  │     │ │                    │   │    │
│  │  └─────┘ └────────────────────┘   │    │
│  │  StatusBar (opaque)               │    │
│  └──────────────────────────────────┘    │
└──────────────────────────────────────────┘

Implementation

Step 1: Enable transparent window

In tauri.conf.json:

{
  "app": {
    "windows": [{
      "transparent": true
    }]
  }
}

Step 2: Make session canvas area transparent

In SessionCanvas.tsx, change the canvas container background:

// Before:
<div className="relative w-full h-full bg-black overflow-hidden">

// After:
<div className="relative w-full h-full bg-transparent overflow-hidden">

Also ensure the <canvas> element itself has no background, or is hidden when the GPU renderer is active.

Step 3: Tell frontend when GPU renderer is active

Add a Tauri command is_gpu_renderer_active() -> bool that checks if Arc<GpuRenderer> exists in state. The frontend uses this to:

  • Hide the <canvas> element (or set display: none)
  • Stop processing IPC frame packets (save CPU)
  • Show "GPU Accelerated" badge in toolbar

Step 4: Disable IPC frame path when GPU active

In session.rs, re-enable the conditional:

if self.shared_frame.is_none() {
    // Only send IPC frames when GPU renderer is NOT active
    let packet = encode_image_update_packet(...);
    self.send_frame_packet(packet)?;
}

Step 5: Verify layer ordering on macOS

The wgpu Metal CAMetalLayer must be BELOW the WKWebView's layer in the NSView hierarchy. Tauri may handle this automatically when transparent: true is set, but verify with:

lldb: po [[window contentView] subviews]

Known risks

Files to modify

  • tauri.conf.json — set transparent: true
  • src/components/session/SessionCanvas.tsx — transparent bg, hide canvas when GPU active
  • src-tauri/src/commands/session.rs — add is_gpu_renderer_active command
  • src-tauri/src/rdp/session.rs — conditional IPC skip
  • src/styles/globals.css — ensure body/root allow transparency

Priority: P0 — this is the gate for the entire GPU pipeline

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestgpuGPU rendering pipeline

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions