feat: dual-modal WASM browser pose estimation demo (ADR-058)

README.md

@@ -75,7 +75,7 @@ docker run -p 3000:3000 ruvnet/wifi-densepose:latest
 |----------|-------------|
 | [User Guide](docs/user-guide.md) | Step-by-step guide: installation, first run, API usage, hardware setup, training |
 | [Build Guide](docs/build-guide.md) | Building from source (Rust and Python) |
-| [Architecture Decisions](docs/adr/README.md) | 48 ADRs — why each technical choice was made, organized by domain (hardware, signal processing, ML, platform, infrastructure) |
+| [Architecture Decisions](docs/adr/README.md) | 49 ADRs — why each technical choice was made, organized by domain (hardware, signal processing, ML, platform, infrastructure) |
 | [Domain Models](docs/ddd/README.md) | 7 DDD models (RuvSense, Signal Processing, Training Pipeline, Hardware Platform, Sensing Server, WiFi-Mat, CHCI) — bounded contexts, aggregates, domain events, and ubiquitous language |
 | [Desktop App](rust-port/wifi-densepose-rs/crates/wifi-densepose-desktop/README.md) | **WIP** — Tauri v2 desktop app for node management, OTA updates, WASM deployment, and mesh visualization |
 
@@ -89,8 +89,12 @@ docker run -p 3000:3000 ruvnet/wifi-densepose:latest
   <em>Real-time pose skeleton from WiFi CSI signals — no cameras, no wearables</em>
   <br>
   <a href="https://ruvnet.github.io/RuView/"><strong>▶ Live Observatory Demo</strong></a>
+  &nbsp;|&nbsp;
+  <a href="https://ruvnet.github.io/RuView/pose-fusion.html"><strong>▶ Dual-Modal Pose Fusion Demo</strong></a>
 
 > The [server](#-quick-start) is optional for visualization and aggregation — the ESP32 [runs independently](#esp32-s3-hardware-pipeline) for presence detection, vital signs, and fall alerts.
+>
+> **Live ESP32 pipeline**: Connect an ESP32-S3 node → run the [sensing server](#sensing-server) → open the [pose fusion demo](https://ruvnet.github.io/RuView/pose-fusion.html) for real-time dual-modal pose estimation (webcam + WiFi CSI). See [ADR-059](docs/adr/ADR-059-live-esp32-csi-pipeline.md).
 
 
 ## 🚀 Key Features

docs/adr/ADR-059-live-esp32-csi-pipeline.md

@@ -0,0 +1,83 @@
+# ADR-059: Live ESP32 CSI Pipeline Integration
+
+## Status
+
+Accepted
+
+## Date
+
+2026-03-12
+
+## Context
+
+ADR-058 established a dual-modal browser demo combining webcam video and WiFi CSI for pose estimation. However, it used simulated CSI data. To demonstrate real-world capability, we need an end-to-end pipeline from physical ESP32 hardware through to the browser visualization.
+
+The ESP32-S3 firmware (`firmware/esp32-csi-node/`) already supports CSI collection and UDP streaming (ADR-018). The sensing server (`wifi-densepose-sensing-server`) already supports UDP ingestion and WebSocket bridging. The missing piece was connecting these components and enabling the browser demo to consume live data.
+
+## Decision
+
+Implement a complete live CSI pipeline:
+
+```
+ESP32-S3 (CSI capture) → UDP:5005 → sensing-server (Rust/Axum) → WS:8765 → browser demo
+```
+
+### Components
+
+1. **ESP32 Firmware** — Rebuilt with native Windows ESP-IDF v5.4.0 toolchain (no Docker). Configured for target network and PC IP via `sdkconfig`. Helper scripts added:
+   - `build_firmware.ps1` — Sets up IDF environment, cleans, builds, and flashes
+   - `read_serial.ps1` — Serial monitor with DTR/RTS reset capability
+
+2. **Sensing Server** — `wifi-densepose-sensing-server` started with:
+   - `--source esp32` — Expect real ESP32 UDP frames
+   - `--bind-addr 0.0.0.0` — Accept connections from any interface
+   - `--ui-path <path>` — Serve the demo UI via HTTP
+
+3. **Browser Demo** — `main.js` updated to auto-connect to `ws://localhost:8765/ws/sensing` on page load. Falls back to simulated CSI if the WebSocket is unavailable (GitHub Pages).
+
+### Network Configuration
+
+The ESP32 sends UDP packets to a configured target IP. If the PC's IP doesn't match the firmware's compiled target, a secondary IP alias can be added:
+
+```powershell
+# PowerShell (Admin)
+New-NetIPAddress -IPAddress 192.168.1.100 -PrefixLength 24 -InterfaceAlias "Wi-Fi"
+```
+
+### Data Flow
+
+| Stage | Protocol | Format | Rate |
+|-------|----------|--------|------|
+| ESP32 → Server | UDP | ADR-018 binary frame (magic `0xC5110001`, I/Q pairs) | ~100 Hz |
+| Server → Browser | WebSocket | ADR-018 binary frame (forwarded) | ~10 Hz (tick-ms=100) |
+| Browser decode | JavaScript | Float32 amplitude/phase arrays | Per frame |
+
+### Build Environment (Windows)
+
+ESP-IDF v5.4.0 on Windows requires:
+- IDF_PATH pointing to the ESP-IDF framework
+- IDF_TOOLS_PATH pointing to toolchain binaries
+- MSYS/MinGW environment variables removed (ESP-IDF rejects them)
+- Python venv from ESP-IDF tools for `idf.py` execution
+
+The `build_firmware.ps1` script handles all of this automatically.
+
+## Consequences
+
+### Positive
+- First end-to-end demonstration of real WiFi CSI → pose estimation in a browser
+- No Docker required for firmware builds on Windows
+- Demo gracefully degrades to simulated CSI when no server is available
+- Same demo works on GitHub Pages (simulated) and locally (live ESP32)
+
+### Negative
+- ESP32 target IP is compiled into firmware; changing it requires a rebuild or NVS override
+- Windows firewall may block UDP:5005; user must allow it
+- Mixed content restrictions prevent HTTPS pages from connecting to ws:// (local only)
+
+## Related
+
+- [ADR-018](ADR-018-esp32-dev-implementation.md) — ESP32 CSI frame format and UDP streaming
+- [ADR-058](ADR-058-ruvector-wasm-browser-pose-example.md) — Dual-modal WASM browser pose demo
+- [ADR-039](ADR-039-edge-intelligence-framework.md) — Edge intelligence on ESP32
+- Issue [#245](https://github.com/ruvnet/RuView/issues/245) — Tracking issue

firmware/esp32-csi-node/build_firmware.ps1

@@ -0,0 +1,31 @@
+# Remove MSYS environment variables that trigger ESP-IDF's MinGW rejection
+Remove-Item env:MSYSTEM -ErrorAction SilentlyContinue
+Remove-Item env:MSYSTEM_CARCH -ErrorAction SilentlyContinue
+Remove-Item env:MSYSTEM_CHOST -ErrorAction SilentlyContinue
+Remove-Item env:MSYSTEM_PREFIX -ErrorAction SilentlyContinue
+Remove-Item env:MINGW_CHOST -ErrorAction SilentlyContinue
+Remove-Item env:MINGW_PACKAGE_PREFIX -ErrorAction SilentlyContinue
+Remove-Item env:MINGW_PREFIX -ErrorAction SilentlyContinue
+
+$env:IDF_PATH = "C:\Users\ruv\esp\v5.4\esp-idf"
+$env:IDF_TOOLS_PATH = "C:\Espressif\tools"
+$env:IDF_PYTHON_ENV_PATH = "C:\Espressif\tools\python\v5.4\venv"
+$env:PATH = "C:\Espressif\tools\xtensa-esp-elf\esp-14.2.0_20241119\xtensa-esp-elf\bin;C:\Espressif\tools\cmake\3.30.2\cmake-3.30.2-windows-x86_64\bin;C:\Espressif\tools\ninja\1.12.1;C:\Espressif\tools\ccache\4.10.2\ccache-4.10.2-windows-x86_64;C:\Espressif\tools\idf-exe\1.0.3;C:\Espressif\tools\python\v5.4\venv\Scripts;$env:PATH"
+
+Set-Location "C:\Users\ruv\Projects\wifi-densepose\firmware\esp32-csi-node"
+
+$python = "$env:IDF_PYTHON_ENV_PATH\Scripts\python.exe"
+$idf = "$env:IDF_PATH\tools\idf.py"
+
+Write-Host "=== Cleaning stale build cache ==="
+& $python $idf fullclean
+
+Write-Host "=== Building firmware (SSID=ruv.net, target=192.168.1.20:5005) ==="
+& $python $idf build
+
+if ($LASTEXITCODE -eq 0) {
+    Write-Host "=== Build succeeded! Flashing to COM7 ==="
+    & $python $idf -p COM7 flash
+} else {
+    Write-Host "=== Build failed with exit code $LASTEXITCODE ==="
+}

firmware/esp32-csi-node/read_serial.ps1

@@ -0,0 +1,14 @@
+$p = New-Object System.IO.Ports.SerialPort('COM7', 115200)
+$p.ReadTimeout = 5000
+$p.Open()
+Start-Sleep -Milliseconds 200
+
+for ($i = 0; $i -lt 60; $i++) {
+    try {
+        $line = $p.ReadLine()
+        Write-Host $line
+    } catch {
+        break
+    }
+}
+$p.Close()

ui/index.html

@@ -29,6 +29,7 @@ <h1>WiFi DensePose</h1>
             <button class="nav-tab" data-tab="applications">Applications</button>
             <button class="nav-tab" data-tab="sensing">Sensing</button>
             <button class="nav-tab" data-tab="training">Training</button>
+            <a href="pose-fusion.html" class="nav-tab" style="text-decoration:none">Pose Fusion</a>
             <a href="observatory.html" class="nav-tab" style="text-decoration:none">Observatory</a>
         </nav>
 

ui/pose-fusion.html

@@ -0,0 +1,160 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="UTF-8">
+  <meta name="viewport" content="width=device-width, initial-scale=1.0">
+  <title>WiFi-DensePose — Dual-Modal Pose Estimation</title>
+  <link rel="stylesheet" href="pose-fusion/css/style.css">
+</head>
+<body>
+
+  <!-- Header -->
+  <header class="header">
+    <div class="header-left">
+      <div class="logo"><span class="pi">&pi;</span> DensePose</div>
+      <div class="header-title">Dual-Modal Pose Estimation — Live Video + WiFi CSI Fusion</div>
+    </div>
+    <div class="header-right">
+      <select id="mode-select" class="mode-select">
+        <option value="dual">Dual Mode (Video + CSI)</option>
+        <option value="video">Video Only</option>
+        <option value="csi">CSI Only (WiFi)</option>
+      </select>
+      <div class="status-badge">
+        <span id="status-dot" class="status-dot offline"></span>
+        <span id="status-label">READY</span>
+      </div>
+      <span id="fps-display" class="fps-badge">-- FPS</span>
+      <a href="index.html" class="back-link">&larr; Dashboard</a>
+      <a href="observatory.html" class="back-link">Observatory &rarr;</a>
+    </div>
+  </header>
+
+  <!-- Main Grid -->
+  <div class="main-grid">
+
+    <!-- Video + Skeleton Panel -->
+    <div class="video-panel">
+      <video id="webcam" autoplay playsinline muted></video>
+      <canvas id="skeleton-canvas"></canvas>
+      <div class="video-overlay-label" id="mode-label">DUAL FUSION</div>
+
+      <div id="camera-prompt" class="camera-prompt">
+        <p>Enable your webcam for live video pose estimation.<br>
+           Or switch to <strong>CSI Only</strong> mode for WiFi-based sensing.</p>
+        <button id="start-camera-btn">Enable Camera</button>
+      </div>
+    </div>
+
+    <!-- Side Panels -->
+    <div class="side-panels">
+
+      <!-- Fusion Confidence -->
+      <div class="panel">
+        <div class="panel-title">&#9670; Fusion Confidence</div>
+        <div class="fusion-bars">
+          <div class="bar-row">
+            <span class="bar-label">Video</span>
+            <div class="bar-track"><div class="bar-fill video" id="video-bar" style="width:0%"></div></div>
+            <span class="bar-value" id="video-bar-val">0%</span>
+          </div>
+          <div class="bar-row">
+            <span class="bar-label">CSI</span>
+            <div class="bar-track"><div class="bar-fill csi" id="csi-bar" style="width:0%"></div></div>
+            <span class="bar-value" id="csi-bar-val">0%</span>
+          </div>
+          <div class="bar-row">
+            <span class="bar-label">Fused</span>
+            <div class="bar-track"><div class="bar-fill fused" id="fused-bar" style="width:0%"></div></div>
+            <span class="bar-value" id="fused-bar-val">0%</span>
+          </div>
+        </div>
+        <div style="margin-top:8px; font-size:10px; color:var(--text-label)">
+          Cross-modal: <span id="cross-modal-sim" style="color:var(--green-glow)">0.000</span>
+        </div>
+      </div>
+
+      <!-- CSI Heatmap -->
+      <div class="panel">
+        <div class="panel-title">&#9670; CSI Amplitude Heatmap</div>
+        <div class="csi-canvas-wrapper">
+          <canvas id="csi-canvas" width="320" height="120"></canvas>
+        </div>
+      </div>
+
+      <!-- Embedding Space -->
+      <div class="panel">
+        <div class="panel-title">&#9670; Embedding Space (2D Projection)</div>
+        <div class="embedding-canvas-wrapper">
+          <canvas id="embedding-canvas" width="320" height="140"></canvas>
+        </div>
+      </div>
+
+      <!-- Latency -->
+      <div class="panel">
+        <div class="panel-title">&#9670; Pipeline Latency</div>
+        <div class="latency-grid">
+          <div class="latency-item">
+            <div class="latency-value" id="lat-video">--</div>
+            <div class="latency-label">Video CNN</div>
+          </div>
+          <div class="latency-item">
+            <div class="latency-value" id="lat-csi">--</div>
+            <div class="latency-label">CSI CNN</div>
+          </div>
+          <div class="latency-item">
+            <div class="latency-value" id="lat-fusion">--</div>
+            <div class="latency-label">Fusion</div>
+          </div>
+          <div class="latency-item">
+            <div class="latency-value" id="lat-total">--</div>
+            <div class="latency-label">Total</div>
+          </div>
+        </div>
+      </div>
+
+      <!-- Controls -->
+      <div class="panel">
+        <div class="panel-title">&#9670; Controls</div>
+        <div class="controls-row">
+          <button class="btn" id="pause-btn">⏸ Pause</button>
+        </div>
+
+        <div class="slider-row">
+          <label>Confidence</label>
+          <input type="range" id="confidence-slider" min="0" max="1" step="0.05" value="0.3">
+          <span class="slider-val" id="confidence-value">0.30</span>
+        </div>
+
+        <div style="margin-top:10px">
+          <div class="panel-title" style="margin-bottom:6px">&#9670; Live CSI Source</div>
+          <div style="display:flex;gap:6px">
+            <input type="text" id="ws-url" placeholder="ws://localhost:3030/ws/csi"
+              style="flex:1;background:rgba(255,255,255,0.05);border:1px solid var(--bg-panel-border);
+                     color:var(--text-primary);padding:5px 8px;border-radius:4px;font-size:11px;
+                     font-family:'JetBrains Mono',monospace">
+            <button class="btn" id="connect-ws-btn">Connect</button>
+          </div>
+        </div>
+      </div>
+
+    </div><!-- /side-panels -->
+
+    <!-- Bottom Bar -->
+    <div class="bottom-bar">
+      <div>
+        WiFi-DensePose &middot; Dual-Modal Pose Estimation &middot;
+        Architecture: MobileNet-V3 &times; 2 &rarr; Attention Fusion &rarr; 17-Keypoint COCO
+      </div>
+      <div>
+        <a href="https://github.com/ruvnet/wifi-densepose">GitHub</a> &middot;
+        CNN: ruvector-cnn (JS fallback) &middot;
+        <a href="observatory.html">Observatory</a>
+      </div>
+    </div>
+
+  </div><!-- /main-grid -->
+
+  <script type="module" src="pose-fusion/js/main.js"></script>
+</body>
+</html>

ui/pose-fusion/build.sh

@@ -0,0 +1,30 @@
+#!/bin/bash
+# Build WASM packages for the dual-modal pose estimation demo.
+# Requires: wasm-pack (cargo install wasm-pack)
+#
+# Usage: ./build.sh
+#
+# Output: pkg/ruvector_cnn_wasm/ — WASM CNN embedder for browser
+
+set -e
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+VENDOR_DIR="$SCRIPT_DIR/../../vendor/ruvector"
+OUT_DIR="$SCRIPT_DIR/pkg/ruvector_cnn_wasm"
+
+echo "Building ruvector-cnn-wasm..."
+wasm-pack build "$VENDOR_DIR/crates/ruvector-cnn-wasm" \
+  --target web \
+  --out-dir "$OUT_DIR" \
+  --no-typescript
+
+# Remove .gitignore so we can commit the build output for GitHub Pages
+rm -f "$OUT_DIR/.gitignore"
+
+echo ""
+echo "Build complete!"
+echo "  WASM: $(du -sh "$OUT_DIR/ruvector_cnn_wasm_bg.wasm" | cut -f1)"
+echo "  JS:   $(du -sh "$OUT_DIR/ruvector_cnn_wasm.js" | cut -f1)"
+echo ""
+echo "Serve the demo: cd $SCRIPT_DIR/.. && python3 -m http.server 8080"
+echo "Open: http://localhost:8080/pose-fusion.html"