[Question] Trained model weights for DensePose inference — how to obtain?

[trumanicu]

Summary

I've successfully set up the full hardware pipeline with an ESP32-S3 sending real CSI data
via UDP, and the sensing server correctly detects presence/motion from the WiFi signals.
However, I cannot get actual human pose estimation working because the trained model weights
are not included in the repository.

What I've verified is working

• ESP32-S3 → UDP broadcast (10 fps, 64 subcarriers, ADR-018 format) ✅
• v1/src/sensing/ws_server.py — real-time presence & motion detection from CSI ✅
• MOCK_HARDWARE=true / MOCK_POSE_DATA=true — mock skeleton renders in UI ✅

The problem

In v1/src/services/pose_service.py (line 112–119), DensePoseHead() is instantiated
with randomly initialized weights — the actual torch.load() call is commented out:

self.densepose_model = DensePoseHead()
# Load model weights if path is provided
# model_state = torch.load(self.settings.pose_model_path)
Without pretrained weights, the model produces random/meaningless pose output, so the
skeleton shown in the Live Demo is not derived from WiFi signals at all.

Questions
Are trained model weights available? If yes, where can I download them and which
config key (POSE_MODEL_PATH) should point to them?

If weights are not available, what dataset and training procedure does this project
expect? Is there a training script (I couldn't find one)? What paired CSI+pose data
format is required?

Is there a roadmap for releasing weights or a pre-collected dataset for training?

Environment
ESP32-S3-DevKitC-1, ESP-IDF v5.x
macOS, Python 3.13, PyTorch 2.10
Confirmed UDP packets: magic=0xC5110001, 148B/frame, 10fps
Any guidance appreciated — the sensing pipeline itself is impressive, I just want to close
the gap to actual pose estimation.

[ruvnet]

Use the rust version. I might remove the python version completely

[trumanicu]

Thanks for the quick reply!

A few follow-up questions before I switch over:

1. ESP32 CSI support — does the Rust sensing server accept ESP32-S3 UDP frames
   in ADR-018 binary format (magic 0xC5110001, 64 subcarriers at 10 fps)?
   The Python ws_server.py has Esp32UdpCollector for this — is there an
   equivalent in the Rust version?

2. Model weights — the original question wasn't answered: does the Rust version
   (wifi-densepose-nn) include trained weights for actual WiFi→pose inference,
   or does it also use random initialization? I want to understand whether the
   skeleton output is derived from real CSI data or is still a demo placeholder.

3. Migration path — is there a quick-start guide for running the Rust sensing
   server with ESP32 hardware? Happy to test and document it.

Context: I have a working ESP32-S3 pipeline (verified 148B/frame UDP at 10fps,
presence/motion detection confirmed). Just trying to close the gap to actual
pose estimation before investing time in the Rust migration.

[IvnSoft]

Yes... the questions were definitely not written by a bot. 🙄

[ruvnet]

Hey, good questions. Honest answer: there are no pretrained WiFi→pose weights in the repo (Rust or Python). The model architectures are there but the weights are random init / placeholders.

Given your setup (ESP32-S3, 64 subcarriers, 10fps UDP — already verified), you're actually in a great position to train your own. That's probably the best path anyway since a model trained on your specific environment (room geometry, AP placement, antenna config) will massively outperform a generic one.

Here's a simple step-by-step outline:

1. Collect paired CSI + ground-truth pose data

• Keep your ESP32 UDP stream running (you've already got this)
• Simultaneously capture ground-truth poses with a camera + off-the-shelf pose estimator (MediaPipe, OpenPose, or even a phone running MoveNet)
• Sync timestamps between CSI frames and camera frames
• Aim for ~30-60 min of varied movements (walking, sitting, standing, reaching, turning)

2. Preprocess & align

• Extract amplitude + phase from CSI subcarriers (the wifi-densepose-signal crate has this)
• Align CSI windows to pose keypoints by timestamp
• Use sliding windows (~1-2 sec) of CSI as input, 17-keypoint skeleton as target

3. Train a lightweight model

• Start simple: 1D CNN or small transformer over CSI time-windows → 17×3 keypoint regression
• The Rust wifi-densepose-train crate has a training pipeline with ruvector integration you can build on
• Or prototype in Python first (faster iteration), then port weights to ONNX for the Rust wifi-densepose-nn backend

4. Environment-specific fine-tuning

• Train/val split by time (not random) to avoid data leakage
• Test in the same room first, then see how it generalizes
• The RuvSense phase_align and coherence modules help normalize across sessions

5. Deploy

• Export to ONNX → load in Rust via the wifi-densepose-nn crate
• The sensing server can serve predictions over WebSocket to the UI

Re: your specific questions — the Rust sensing server does handle ESP32 UDP frames (see wifi-densepose-hardware crate, ESP32 aggregator + TDM protocol). For migration, the wifi-densepose-cli is the quickest entry point.

The key insight is that WiFi→pose is extremely environment-dependent, so a model trained in your room with your hardware will always beat a generic pretrained one. Think of it like training a custom object detector — the architecture is standard, but the data needs to be yours.

[ruvnet]

About Model Weights and DensePose Inference

Great investigation! You've correctly identified the core situation.

Current State

The repository provides:

1. Working WiFi sensing pipeline - CSI frame parsing, signal processing, presence/motion detection ✅
2. DensePose neural network architecture - The model structure is defined
3. Training pipeline scaffolding - Dataset loaders, training loops, evaluation

What's NOT included: Pre-trained model weights for the WiFi→DensePose inference.

Why No Pre-Trained Weights?

1. No public paired CSI+Pose datasets exist that match our ADR-018 frame format
2. Training requires hardware setup - Multiple synchronized ESP32 nodes + camera for ground truth
3. Domain gap - WiFi CSI is highly environment-specific; weights trained in one room may not generalize

How the System is Designed

The architecture follows a staged approach:

1. Stage 1 (Working Now): WiFi sensing features → Presence/Motion/Breathing detection

   • Uses signal processing, no neural network required
   • v1/src/sensing/ws_server.py demonstrates this

2. Stage 2 (Training Required): WiFi features → Body keypoints

   • Requires paired CSI+skeleton ground truth data
   • The DensePoseHead expects these trained weights

3. Stage 3: Keypoints → Full DensePose surface estimation

   • Future work, depends on Stage 2

To Train Your Own Model

1. Collect paired data: Record CSI frames while capturing skeleton keypoints (camera + pose estimation like MediaPipe or OpenPose)

2. Format as MM-Fi: See docs/adr/ADR-015-mmfi-wipose-training-datasets.md for the expected data format

3. Run training:

   cd v1
   python -m src.training.train_pose --dataset /path/to/paired_data --epochs 100

4. Set model path:

   # In config or environment
   POSE_MODEL_PATH=/path/to/trained_weights.pt

Alternative: Use Sensing-Only Features

If you just need presence/motion/breathing detection (not full body pose), the sensing pipeline works today without neural weights:

# The WebSocket server at ws://localhost:3001/ws/sensing
# streams real-time sensing data including:
{
    "presence_detected": true,
    "motion_level": 0.42,
    "breathing_rate": 16.2,  # BPM
    "signal_quality": 0.85
}

Roadmap

We're working on:

• Public paired CSI+Pose dataset (ADR-015)
• Pre-trained checkpoint for common indoor environments
• Transfer learning guide for custom environments

Thanks for your detailed analysis - would you be interested in contributing to the training pipeline or data collection?