Skip to content

Use G1 WebRTC camera in headless setup#1553

Open
kaiknower wants to merge 9 commits intodimensionalOS:devfrom
kaiknower:fix-g1-headless-camera
Open

Use G1 WebRTC camera in headless setup#1553
kaiknower wants to merge 9 commits intodimensionalOS:devfrom
kaiknower:fix-g1-headless-camera

Conversation

@kaiknower
Copy link

Summary

  • publish G1 camera image and camera info directly from the WebRTC connection
  • remove the local webcam dependency from the headless G1 primitive blueprint
  • keep the existing G1 camera TF publication so perception modules still receive camera frames in the expected coordinate system

Motivation

unitree-g1 was trying to open /dev/video0 on the deployment machine via CameraModule, which breaks headless setups where the camera is actually on the G1 robot.

Validation

  • verified Python syntax with python -m py_compile on the modified files
  • manually checked that the blueprint no longer instantiates the local webcam path

spomichter and others added 9 commits January 23, 2026 07:34
…lease v0.0.8: Unitree Go2 Navigation & Exploration Beta

Pre-Release v0.0.8: Unitree Go2 Navigation & Exploration Beta, Transport Updates, Documentation updates, Rerun fixes, Person follow, Readme updates

## What's Changed
* Small docs clarification about stream getters by @leshy in dimensionalOS#1043
* Fix split view on wide monitors by @jeff-hykin in dimensionalOS#1048
* Docs: Install & Develop  by @jeff-hykin in dimensionalOS#1022
* Add uv to nix and fix resulting problems by @jeff-hykin in dimensionalOS#1021
* v0.0.8 by @paul-nechifor in dimensionalOS#1050
* Style changes in docs by @paul-nechifor in dimensionalOS#1051
* Revert "Add uv to nix and fix resulting problems" by @leshy in dimensionalOS#1053
* Transport benchmarks + Raw ros transport by @leshy in dimensionalOS#1038
* feat: default to rerun-web and auto-open browser on startup (browser … by @Nabla7 in dimensionalOS#1019
* bbox detections visual check by @leshy in dimensionalOS#1017
* fix: only auto-open browser for rerun-web viewer backend by @Nabla7 in dimensionalOS#1066
* move slow tests to integration by @paul-nechifor in dimensionalOS#1063
* Streamline transport start/stop methods by @Kaweees in dimensionalOS#1062
* Person follow skill with EdgeTAM by @paul-nechifor in dimensionalOS#1042
* fix: increase costmap floor z_offset to avoid z-fighting by @Nabla7 in dimensionalOS#1073
* Fixed issue dimensionalOS#1074 by @alexlin2 in dimensionalOS#1075
* ROS transports initial by @leshy in dimensionalOS#1057
* Fix System Config Values for LCM on MacOS and Refactor by @jeff-hykin in dimensionalOS#1065
* SHM Transport basic fixes by @leshy in dimensionalOS#1041
* commented out Mem Transport test case by @leshy in dimensionalOS#1077
* Docs/advanced streams update 2 by @leshy in dimensionalOS#1078
* Fix more tests by @paul-nechifor in dimensionalOS#1071
* feat: navigation docker updates from bona_local_dev by @baishibona in dimensionalOS#1081
* Fix missing dependencies by @Kaweees in dimensionalOS#1085
* Release readme fixes by @spomichter in dimensionalOS#1076

## New Contributors
* @baishibona made their first contribution in dimensionalOS#1081

**Full Changelog**: dimensionalOS/dimos@v0.0.7...v0.0.8
…ne change to HTTPS from SSH, get_data change, LFS changes

v0.0.9 Release Patch: Git clone change to HTTPS from SSH, get_data change, LFS changes
Release v0.0.10: Manipulation Stack, MuJoCo Simulation, DDS Transport, Web and Native Visualization via Rerun


## Highlights

88+ commits, 20 contributors, 700+ files changed.

The TLDR: **a complete manipulation stack**, **MuJoCo simulation**, **DDS transport**, and **a rewritten visualization pipeline**. Agents are no longer bolted on top — they're refactored as native modules with direct stream access. The entire ROS message dependency has been removed from core DimOS, and we've added VR, phone, and arm teleoperation stacks. You can now vibecode a pick-and-place task from natural language to motor commands. Installation has been significantly streamlined — no more direnv, simpler setup, and the web viewer is now the default.

---

## 🚀 New Features

### Simulation
- **MuJoCo simulation module** — Run any DimOS blueprint in simulation with no hardware. Supports xArm and Unitree embodiments, parses MJCF/URDF for robot properties, monotonic clock timing (no `time.sleep`). `dimos --simulation run unitree-go2` ([dimensionalOS#1035](dimensionalOS#1035)) by @jca0
- **Simulation teleop blueprints** — Added simulation teleop blueprints for Piper, xArm6, and xArm7. ([dimensionalOS#1308](dimensionalOS#1308)) by @mustafab0

### Manipulation
- **Modular manipulation stack** — Full planning stack with Drake: FK/IK solvers (Jacobian + Drake optimization), RRT path planning, world model with obstacle monitoring, multi-robot management. xArm6/7 and Piper support. ([dimensionalOS#1079](dimensionalOS#1079)) by @mustafab0
- **Joint servo and cartesian controllers** — Joint position/velocity controllers and cartesian IK task with Pinocchio solver. PoseStamped stream input for real-time control. ([dimensionalOS#1116](dimensionalOS#1116)) by @mustafab0
- **GraspGen integration** — Grasp generation via Docker-hosted GPU model. Lazy container startup, thread-safe init, RPC `generate_grasps()` returns ranked PoseArray. ([dimensionalOS#1119](dimensionalOS#1119), [dimensionalOS#1234](dimensionalOS#1234)) by @JalajShuklaSS
- **Gripper control** — Gripper RPC methods on control coordinator, exposed adapter property for custom implementations. ([dimensionalOS#1213](dimensionalOS#1213)) by @mustafab0
- **Detection3D and Object support** — Object input topics, TF support on manipulation module, pointcloud-to-convex-hull for Drake imports. ([dimensionalOS#1236](dimensionalOS#1236)) by @mustafab0
- **Agentic pick and place** — Reimplemented manipulation skills for agent-driven pick-and-place workflows. ([dimensionalOS#1237](dimensionalOS#1237)) by @mustafab0

### Teleoperation
- **Quest VR teleoperation** — Full WebXR + Deno bridge stack. Quest controller data (pose, trigger, grip) streamed to DimOS modules. Monitor-style locking for control loops. ([dimensionalOS#1215](dimensionalOS#1215)) by @ruthwikdasyam
- **Phone teleoperation** — Control Go2 from your phone with a web-based teleop interface. ([dimensionalOS#1280](dimensionalOS#1280)) by @ruthwikdasyam
- **Arm teleop with Pinocchio IK** — Single and dual arm teleoperation using Pinocchio inverse kinematics. Blueprints for xArm, Piper, and dual configurations. ([dimensionalOS#1246](dimensionalOS#1246)) by @ruthwikdasyam

### Transports & Infrastructure
- **DDS transport protocol** — CycloneDDS transport with configurable QoS (high-throughput and reliable profiles). Optional install, benchmark integration. ([dimensionalOS#1174](dimensionalOS#1174)) by @Kaweees
- **Pubsub pattern subscriptions** — Glob and regex pattern matching for topic subscriptions. `subscribe_all()` for bridge-style consumers. Topic type encoding in channel strings (`/topic#module.ClassName`). ([dimensionalOS#1114](dimensionalOS#1114)) by @leshy
- **LCM raw bytes passthrough** — Skip `lcm_encode()` when message is already bytes. ([dimensionalOS#1223](dimensionalOS#1223)) by @leshy
- **Unified TimeSeriesStore** — Pluggable backends (InMemory, SQLite, Pickle, PostgreSQL) with SortedKeyList for O(log n) operations. Replaces the old replay system and TimestampedCollection. Collection API with slice, range, and streaming methods. ([dimensionalOS#1080](dimensionalOS#1080)) by @leshy
- **DimosROS benchmark tests** — Benchmark suite for ROS transport performance. ([dimensionalOS#1087](dimensionalOS#1087)) by @leshy

### Navigation
- **FASTLIO2 support** — Hardware-verified localization with arm64 support. Docker deployment with FAR Planner, terrain analysis, and bagfile playback mode. Builds or-tools from source on arm64. ([dimensionalOS#1149](dimensionalOS#1149)) by @baishibona
- **Native Livox + FASTLIO2 module** — First-class DimOS native module for Livox Mid-360 lidar with FASTLIO2 localization. ([dimensionalOS#1235](dimensionalOS#1235)) by @leshy

### Visualization
- **RerunBridge module and CLI** — New bridge that subscribes to all LCM messages and logs those with `to_rerun()` to Rerun viewer. GlobalConfig singleton, web viewer support. Replaces the old rerun initialization system. ([dimensionalOS#1154](dimensionalOS#1154)) by @leshy
- **Webcam rerun visualization** — Camera module logs to Rerun with pinhole projection for 3D visualization. ([dimensionalOS#1117](dimensionalOS#1117)) by @ruthwikdasyam
- **Default viewer switched to rerun-web** — Browser-based viewer is now the default for broader compatibility. No native viewer install needed. ([dimensionalOS#1324](dimensionalOS#1324)) by @spomichter

### Agents
- **Agent refactor** — Restructured agent module with cleaner imports and global config integration. ([dimensionalOS#1211](dimensionalOS#1211)) by @paul-nechifor
- **Timestamp knowledge** — Agents now have timestamp awareness in prompts for temporal reasoning. ([dimensionalOS#1093](dimensionalOS#1093)) by @ClaireBookworm
- **Observe skill** — Go2 can now observe (capture and describe) its environment via agent skill. ([dimensionalOS#1109](dimensionalOS#1109)) by @paul-nechifor

### Platform & Hardware
- **G1 without ROS** — Unitree G1 blueprints decoupled from ROS dependency. Lazy imports for fast startup. ([dimensionalOS#1221](dimensionalOS#1221)) by @jeff-hykin
- **ARM (aarch64) support** — DimOS runs on ARM hardware. Platform-conditional dependencies, open3d source builds for arm64. ([dimensionalOS#1229](dimensionalOS#1229)) by @jeff-hykin
- **Universal joint/hardware schema** — `HardwareComponent` dataclass with `JointState`, `JointName` type aliases. Backend registry with auto-discovery for SDK adapters. ([dimensionalOS#1040](dimensionalOS#1040), [dimensionalOS#1067](dimensionalOS#1067)) by @mustafab0

---

## 🔧 Improvements

- **Optional Dask** — Start without Dask using `--no-dask` flag. Startup time reduced from ~60s to ~45s. ([dimensionalOS#1111](dimensionalOS#1111), [dimensionalOS#1232](dimensionalOS#1232)) by @paul-nechifor
- **RPC rework** — Renamed `ModuleBlueprint` → `_BlueprintAtom`, `ModuleBlueprintSet` → `Blueprint`, `ModuleConnection` → `Stream`. Added `ModuleRef`, improved type hints throughout. ([dimensionalOS#1143](dimensionalOS#1143)) by @jeff-hykin
- **Image class simplification** — Rewritten as pure NumPy dataclass. Removed CUDA backend, unused methods (solve_pnp, csrt_tracker), and image_impls/ directory. ([dimensionalOS#1161](dimensionalOS#1161)) by @leshy
- **Odometry message cleanup** — Simplified Odometry message type. ([dimensionalOS#1256](dimensionalOS#1256)) by @leshy
- **Remove all ROS message dependencies** — Purged ROS message types from core DimOS. Refactored rosnav to use ROSTransport. Removed dead ROS bridge code. ([dimensionalOS#1230](dimensionalOS#1230)) by @alexlin2
- **Removed bad function serialization** — Eliminated unnecessary serialization of Python functions. ([dimensionalOS#1121](dimensionalOS#1121)) by @paul-nechifor
- **Benchmark IEC units** — Switched bandwidth benchmarks from SI to IEC units for accuracy. ([dimensionalOS#1147](dimensionalOS#1147)) by @leshy
- **Pubsub typing improvements** — Thread-safety locks on `subscribe_new_topics` and `subscribe_all`. Proper type params across pubsub stack. ([dimensionalOS#1153](dimensionalOS#1153)) by @leshy
- **Autogenerated blueprint list** — Blueprints are now auto-discovered and listed. ([dimensionalOS#1100](dimensionalOS#1100)) by @paul-nechifor
- **Generic Buttons message** — Renamed `QuestButtons` to `Buttons` with generic field names for cross-platform teleop. ([dimensionalOS#1261](dimensionalOS#1261)) by @ruthwikdasyam
- **Dev container uses ros-dev image** — `./bin/dev` now runs the ROS-enabled dev image. ([dimensionalOS#1170](dimensionalOS#1170)) by @leshy
- **LSP support** — Added python-lsp-server and python-lsp-ruff to dev dependencies. ([dimensionalOS#1169](dimensionalOS#1169)) by @leshy
- **Lazy-load pyrealsense2** — RealSense camera module uses lazy imports to avoid errors in simulation environments without the SDK. ([dimensionalOS#1309](dimensionalOS#1309)) by @spomichter
- **Removed unused mmcv and mmengine** — Dead Detic dependencies removed, eliminating slow source builds from install. ([dimensionalOS#1319](dimensionalOS#1319)) by @spomichter
- **Simplified installation** — Removed direnv requirement, streamlined install instructions across all platforms. ([dimensionalOS#1315](dimensionalOS#1315)) by @spomichter
- **DDS extra excluded from --all-extras** — `cyclonedds` requires a source build, so `dds` is now excluded from `uv sync --all-extras` by default. ([dimensionalOS#1318](dimensionalOS#1318)) by @spomichter
- **Nix pre-commit skip** — Skip pre-commit install if hooks already exist. ([dimensionalOS#1162](dimensionalOS#1162)) by @leshy
- **Removed base-requirements** — Consolidated dependency management. ([dimensionalOS#1098](dimensionalOS#1098)) by @paul-nechifor
- **Removed old graspnet** — Cleaned up deprecated graspnet version. ([dimensionalOS#1248](dimensionalOS#1248)) by @paul-nechifor
- **Code cleanup** — Removed `tofix` markers ([dimensionalOS#1216](dimensionalOS#1216)), fixed ruff issues ([dimensionalOS#1112](dimensionalOS#1112)), removed old README_installation.md ([dimensionalOS#1101](dimensionalOS#1101)) by @paul-nechifor

---

## 🐛 Bug Fixes

- Fix LFS updating (move from .local to venv) ([dimensionalOS#1090](dimensionalOS#1090)) by @jeff-hykin
- Launch hotfixes: git clone HTTPS, get_data main branch ([dimensionalOS#1091](dimensionalOS#1091)) by @spomichter
- Fix camera demo not showing in Rerun ([dimensionalOS#1148](dimensionalOS#1148)) by @jeff-hykin
- Default to rerun native viewer ([dimensionalOS#1099](dimensionalOS#1099)) by @Nabla7
- Fix exploration blocking agent loop ([dimensionalOS#1258](dimensionalOS#1258)) by @paul-nechifor
- Fix person-follow blocking agent loop ([dimensionalOS#1278](dimensionalOS#1278)) by @paul-nechifor
- Skip metric3d tests on unsupported xformers GPUs (Blackwell compute capability >9.0) ([dimensionalOS#1225](dimensionalOS#1225)) by @leshy
- Fix manipulation tests ([dimensionalOS#1218](dimensionalOS#1218), [dimensionalOS#1247](dimensionalOS#1247)) by @jeff-hykin, @paul-nechifor
- Fix control coordinator e2e test ([dimensionalOS#1212](dimensionalOS#1212)) by @mustafab0
- Fix xarm7-sim broken e2e tests ([dimensionalOS#1294](dimensionalOS#1294)) by @paul-nechifor
- Pin langchain to restore supported providers ([dimensionalOS#1241](dimensionalOS#1241)) by @spomichter
- Fix missing library dependencies in Nix flake ([dimensionalOS#1240](dimensionalOS#1240)) by @Kaweees
- Fix discord invite link ([dimensionalOS#1122](dimensionalOS#1122)) by @spomichter
- macOS edgecase fix ([dimensionalOS#1096](dimensionalOS#1096)) by @jeff-hykin
- Fix second N in logo ([dimensionalOS#1250](dimensionalOS#1250)) by @jeff-hykin
- Fix Unitree Go2 minor issues ([dimensionalOS#1307](dimensionalOS#1307)) by @paul-nechifor
- Fix broken tests ([dimensionalOS#1305](dimensionalOS#1305)) by @ruthwikdasyam
- Fix `uv sync` for some macOS systems ([dimensionalOS#1322](dimensionalOS#1322)) by @jeff-hykin
- Fix mmcv install ([dimensionalOS#1313](dimensionalOS#1313)) by @paul-nechifor
- Fix mypy issues ([dimensionalOS#1150](dimensionalOS#1150), [dimensionalOS#1167](dimensionalOS#1167), [dimensionalOS#1257](dimensionalOS#1257)) by @leshy, @paul-nechifor, @jeff-hykin
- Fix Nix install uv pip extras ([dimensionalOS#1321](dimensionalOS#1321)) by @spomichter

---

## 📚 Documentation

- **Major docs overhaul** — New README with feature grid, hardware table, quickstart. Navigation, transports, data streams, and agent docs. ([dimensionalOS#1295](dimensionalOS#1295)) by @leshy
- **Day 1 docs** — Comprehensive getting started guide, development docs, contributing guide, architecture overview. Executable blueprint docs via md-babel-py. ([dimensionalOS#1064](dimensionalOS#1064)) by @jeff-hykin
- **Arm integration guide** — How-to for integrating new robotic arms with DimOS. ([dimensionalOS#1238](dimensionalOS#1238)) by @mustafab0
- **MCP documentation update** — Updated MCP install and usage instructions. ([dimensionalOS#1251](dimensionalOS#1251)) by @Kaweees
- **Docker docs** — First pass on Docker deployment documentation. ([dimensionalOS#1151](dimensionalOS#1151)) by @leshy
- **Transports documentation** — Encode/decode mixins, SHM examples, ROS/DDS transport docs. ([dimensionalOS#1107](dimensionalOS#1107)) by @leshy
- **Rerun API examples** — Updated examples for the new RerunBridge API. ([dimensionalOS#1262](dimensionalOS#1262)) by @jeff-hykin
- **PR template** added ([dimensionalOS#1172](dimensionalOS#1172)) by @christiefhyang
- **Simplified install instructions** — Removed direnv, streamlined across all platforms. ([dimensionalOS#1315](dimensionalOS#1315)) by @spomichter
- **Python example restored** — Added back the Python usage example. ([dimensionalOS#1317](dimensionalOS#1317)) by @jeff-hykin
- **Nix install updated** — Replaced uv with pip for Nix compatibility. ([dimensionalOS#1326](dimensionalOS#1326)) by @ruthwikdasyam
- **README improvements** ([dimensionalOS#1311](dimensionalOS#1311)) by @paul-nechifor
- **Simplified writing docs** — Consolidated writing_docs to a single markdown file. ([dimensionalOS#1254](dimensionalOS#1254)) by @jeff-hykin

---

## 🏗️ CI & Build

- **ci-complete gate** — Dynamic branch protection via single aggregated status check. MD-only PRs no longer blocked. ([dimensionalOS#1279](dimensionalOS#1279)) by @spomichter
- **Path-based test filtering** — Test jobs fully skip (no container spin-up) when no relevant code changed. ([dimensionalOS#1284](dimensionalOS#1284), [dimensionalOS#1286](dimensionalOS#1286)) by @spomichter
- **Navigation docker build workflow** — CI builds for the ROS navigation stack. ([dimensionalOS#1259](dimensionalOS#1259)) by @spomichter
- **CUDA test marker** — `@pytest.mark.cuda` for GPU-dependent tests. ([dimensionalOS#1220](dimensionalOS#1220)) by @jeff-hykin
- **e2e test marker** — Marked end-to-end tests for selective CI runs. ([dimensionalOS#1110](dimensionalOS#1110)) by @paul-nechifor
- **pytest stdin fix** — Added `-s` to default addopts for LCM autoconf compatibility. ([dimensionalOS#1320](dimensionalOS#1320)) by @spomichter

---

## ⚠️ Breaking Changes

- **RPC renames**: `ModuleBlueprint` → `_BlueprintAtom`, `ModuleBlueprintSet` → `Blueprint`, `ModuleConnection` → `Stream` ([dimensionalOS#1143](dimensionalOS#1143))
- **Image class rewrite**: `CudaImage` and `NumpyImage` removed. Image is now a pure NumPy dataclass. Methods like `solve_pnp`, `csrt_tracker`, `from_depth`, `to_depth_meters` removed. ([dimensionalOS#1161](dimensionalOS#1161))
- **ROS messages removed from core**: All `to_ros`/`from_ros` conversion methods removed. Use `ROSTransport` instead. ([dimensionalOS#1230](dimensionalOS#1230))
- **QuestButtons → Buttons**: Renamed with generic field names. ([dimensionalOS#1261](dimensionalOS#1261))
- **RerunBridge replaces old rerun init**: `dimos.dashboard.rerun_init` removed. Use `RerunBridgeModule` or the `rerun-bridge` CLI. ([dimensionalOS#1154](dimensionalOS#1154))
- **Unitree directory restructuring**: `unitree_go2` → `unitree/go2`, `unitree_g1` → `unitree/g1`. Blueprint names updated. ([dimensionalOS#1221](dimensionalOS#1221))
- **Default viewer is now rerun-web**: Use `--viewer-backend rerun` to restore native viewer. ([dimensionalOS#1324](dimensionalOS#1324))

---

## Quickstart

```bash
# Install
uv pip install dimos[base,unitree]

# Try it (no hardware needed)
# NOTE: First run downloads ~2.4 GB from LFS
dimos --replay run unitree-go2

# Simulate
uv pip install dimos[base,unitree,sim]
dimos --simulation run unitree-go2
```

---

## New Contributors 🎉

- @ruthwikdasyam — Quest VR teleoperation, phone teleop, arm teleop, webcam rerun viz
- @JalajShuklaSS — GraspGen integration
- @jca0 — MuJoCo simulation module
- @christiefhyang — PR template

---

**Full Changelog**: [v0.0.9...v0.0.10](dimensionalOS/dimos@v0.0.9...v0.0.10)
Release v0.0.11

82 PRs, 10 contributors, 396 files changed.

This release brings a production CLI, MCP tooling, temporal memory, and first-class support for coding agents. Dask has been removed. The entire stack now runs from `dimos run` through `dimos stop`.

### Agent-Native Development

DimOS is now built to be driven by coding agents. Point OpenClaw, Claude Code, or Cursor at [AGENTS.md](AGENTS.md) and they can build, run, and debug Dimensional applications using the CLI and MCP interfaces directly.

- **AGENTS.md** — comprehensive onboarding doc: architecture, CLI reference, skill rules, blueprint quick-reference. Your agent reads this and starts coding.
- **MCP server** — all `@skill` methods exposed as HTTP tools. External agents call `dimos mcp call relative_move --arg forward=0.5` or connect via JSON-RPC.
- **MCP CLI** — `dimos mcp list-tools`, `dimos mcp call`, `dimos mcp status`, `dimos mcp modules`
- **Agent context logging** — MCP tool calls and agent messages logged to per-run JSONL for debugging and replay.

### CLI & Daemon

Full process lifecycle — no more Ctrl-C in tmux.

- `dimos run --daemon` — background execution with health checks and run registry
- `dimos stop [--force]` — graceful shutdown with SIGTERM → SIGKILL fallback
- `dimos restart` — replays the original CLI arguments
- `dimos status` — PID, blueprint, uptime, MCP port
- `dimos log -f` — structured per-run logs with follow, JSON output, filtering
- `dimos show-config` — resolved GlobalConfig with source tracing

### Temporal-Spatial Memory

Robots in physical space ingest hours of video and lidar. Temporal-spatial memory gives them a human-like understanding of the world — causal object relationships, entity tracking through time and physical space, and the ability to answer complex temporal queries:

*Who spends the most time in the kitchen? What time on average do I wake up? Which set of switches toggles the main lights? Who was at the office at 9am last Thursday?*

Traditional frame-level embeddings (CLIP, ViT) lose temporal context and don't scale beyond a handful of frames. Video transformers are expensive and don't operate in RGB-D. Dimensional agents work with video + lidar natively, tracking entities across hours and days.

```bash
dimos --replay --replay-dir unitree_go2_office_walk2 run unitree-go2-temporal-memory
```

### Interactive Viewer

Custom Rerun fork (`dimos-viewer`) is now the default. Click-to-navigate: click a point in the 3D view → PointStamped → A* planner → robot moves.

- Camera | 3D split layout on Go2, G1, and drone blueprints
- Native keyboard teleop in the viewer
- `--viewer rerun|rerun-web|rerun-connect|foxglove|none`

### Drone Support

Drone blueprints modernized to match Go2 composition pattern. `drone-basic` and `drone-agentic` work with replay, Rerun, and the full CLI.

```bash
dimos --replay run drone-basic
dimos --replay run drone-agentic
```

### More

- **Go2 fleet control** — multi-robot with `--robot-ips` (dimensionalOS#1487)
- **Replay `--replay-dir`** — select dataset, loops by default (dimensionalOS#1519, dimensionalOS#1494)
- **Interactive install** — `curl -fsSL .../install.sh | bash` (dimensionalOS#1395)
- **Nix on non-Debian Linux** (dimensionalOS#1472)
- **Remove Dask** — native worker pool (dimensionalOS#1365)
- **Remove asyncio dependency** (dimensionalOS#1367)
- **Perceive loop** — continuous observation module for agents (dimensionalOS#1411)
- **Worker resource monitor** — `dtop` TUI (dimensionalOS#1378)
- **G1 agent wiring fix** (dimensionalOS#1518)
- **Rerun rate limiting** — prevents viewer OOM on continuous streams (dimensionalOS#1509, dimensionalOS#1521)
- **RotatingFileHandler** — prevents unbounded log growth (dimensionalOS#1492)
- **Test coverage** (dimensionalOS#1397), draft PR CI skip (dimensionalOS#1398), manipulation test fixes (dimensionalOS#1522)

### Breaking Changes

- `--viewer-backend` renamed to `--viewer`
- Dask removed — blueprints using Dask workers need migration to native worker pool
- Default viewer changed from `rerun-web` to `rerun` (native dimos-viewer)

### Contributors

@spomichter, @PaulNechifor, @ruthwikdasyam, @summeryang, @MustafaBhadsorawala, @leshy, @sambull, @JeffHykin, @RadientBrain

## Contributor License Agreement

- [x] I have read and approved the [CLA](https://github.com/dimensionalOS/dimos/blob/main/CLA.md).
docs: README additions: temporal-memory fix, formatting (dimensionalOS#1535)
@greptile-apps
Copy link
Contributor

greptile-apps bot commented Mar 14, 2026

Greptile Summary

This PR moves camera image and metadata publishing from the blueprint layer (via CameraModule + local webcam) into the G1Connection class itself, sourcing video frames directly from the robot's WebRTC stream. This fixes headless deployments where /dev/video0 doesn't exist on the deployment machine.

  • connection.py: G1Connection now publishes color_image from the WebRTC video stream and runs a 1Hz background thread for camera_info + TF transforms (camera_link and camera_optical), matching the pattern already established by G1SimConnection.
  • uintree_g1_primitive_no_nav.py: Removes _create_webcam(), _camera variable, and all related imports (Webcam, camera_module, zed, geometry types). The blueprint already had color_image and camera_info wired via LCM transports, so downstream consumers are unaffected.
  • The stop() method was also improved to guard against self.connection being None, making it more robust than the previous assert-based approach.

Confidence Score: 4/5

  • This PR is safe to merge — it applies a well-established pattern from G1SimConnection to the real hardware connection.
  • The changes are focused and well-motivated. The new camera publishing in G1Connection closely follows the existing G1SimConnection pattern. The blueprint cleanup is clean with no orphaned references. One minor style concern about a class-level mutable default, but it's non-blocking since the object is never mutated in place.
  • dimos/robot/unitree/g1/connection.py — verify the camera_info_static class-level default is intentional

Important Files Changed

Filename Overview
dimos/robot/unitree/g1/connection.py Adds WebRTC camera image publishing and a 1Hz camera_info/TF publishing loop to G1Connection. The implementation mirrors the existing G1SimConnection pattern and the old CameraModule behavior. One minor style note about a class-level mutable default.
dimos/robot/unitree/g1/blueprints/primitive/uintree_g1_primitive_no_nav.py Removes local webcam/CameraModule setup and related imports. Camera streams now come from G1Connection via LCM transports, which were already wired. Clean removal with no orphaned references.

Sequence Diagram

sequenceDiagram
    participant G1 as G1 Robot (WebRTC)
    participant Conn as G1Connection
    participant LCM as LCM Transport
    participant BP as Blueprint / Consumers

    Note over Conn: start() called
    Conn->>G1: Open WebRTC connection
    G1-->>Conn: video_stream() Observable

    loop Every video frame
        G1->>Conn: Video frame (Image)
        Conn->>Conn: color_image.publish(image)
        Conn->>LCM: Broadcast /color_image
        LCM->>BP: Image delivered
    end

    loop Every 1 second (background thread)
        Conn->>Conn: camera_info.publish(static_info.with_ts)
        Conn->>LCM: Broadcast /camera_info
        Conn->>Conn: tf.publish(camera_link, camera_optical)
        LCM->>BP: CameraInfo + TF delivered
    end

    Note over Conn: stop() called
    Conn->>Conn: _stop_event.set()
    Conn->>G1: connection.stop()
    Conn->>Conn: Join camera_info thread
Loading

Last reviewed commit: d620ace

ip: str | None
connection_type: str | None = None
_global_config: GlobalConfig
camera_info_static: CameraInfo = _camera_info_static()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Class-level mutable default is shared across instances

_camera_info_static() is called once at class-definition time, so every G1Connection instance shares the same CameraInfo object. This is safe today because with_ts() returns a new copy and the static object is never mutated, but it's a subtle footgun if the object is ever modified in-place later. For comparison, G1SimConnection reads camera info at runtime from self.connection.camera_info_static.

Consider initializing this in __init__ instead:

Suggested change
camera_info_static: CameraInfo = _camera_info_static()
camera_info_static: CameraInfo

And then in __init__, add self.camera_info_static = _camera_info_static().

@spomichter spomichter changed the base branch from main to dev March 14, 2026 11:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants