diff --git a/README.md b/README.md index 1662444..5aaeb06 100644 --- a/README.md +++ b/README.md @@ -1,255 +1,138 @@

LuckyLab

- A unified robot learning framework powered by LuckyEngine + RL and IL training framework for LuckyEngine

+ Lucky Robots + LuckyEngine License: MIT - Python 3.10+ +

+

+ Python 3.10+ + PyTorch >= 2.0 + luckyrobots >= 0.1.84 Ruff

-LuckyLab is a modular, config-driven framework that brings reinforcement learning, imitation learning, and real-time visualization together in one place. It communicates with LuckyEngine through [luckyrobots](https://github.com/luckyrobots/luckyrobots) and runs on both CPU and GPU. - -The framework ships with locomotion and manipulation tasks but is easily extensible to any robot or task. It supports all imitation learning algorithms in [LeRobot](https://github.com/huggingface/lerobot) and multiple RL algorithms via [skrl](https://github.com/Toni-SM/skrl) and [Stable Baselines3](https://github.com/DLR-RM/stable-baselines3). Live inspection is available through [Rerun](https://rerun.io) and [Viser](https://github.com/nerfstudio-project/viser). - -| Robot | Task | Learning | -|-------|------|----------| -| Unitree Go2 | Velocity tracking | RL (PPO, SAC, TD3, DDPG) | -| Piper | Pick-and-place | IL (via LeRobot) | +LuckyLab is the training and inference layer for robots simulated in [LuckyEngine](https://github.com/luckyrobots/LuckyEngine). It connects to LuckyEngine over gRPC (via the [luckyrobots](https://github.com/luckyrobots/luckyrobots) client), sends joint-level actions, and receives observations each step — all physics and rendering runs in LuckyEngine. --- - -## Requirements - -- Python 3.10+ -- [LuckyEngine](https://luckyrobots.com) executable -- [luckyrobots](https://github.com/luckyrobots/luckyrobots) >= 0.1.81 -- PyTorch >= 2.0 - -## Installation +## Quick Start +### 1. Installation ```bash -git clone https://github.com/luckyrobots/luckylab.git +git clone -b mick/release-2026-1 --single-branch https://github.com/luckyrobots/luckylab.git cd luckylab -# Core + RL -uv sync --group rl - -# Core + IL (LeRobot) -uv sync --group il - -# Everything (RL + IL + Rerun + dev tools) -uv sync --all-groups +# Run the setup script for your OS +./setup.bat # Windows +./setup.sh # Linux ``` +### 2. Prepare LuckyEngine ---- - -## Quick Start - -### Train +1. Launch LuckyEngine +2. Download the Piper Block Stacking project +3. Open the Piper Block Stacking scene +4. Open the gRPC Panel +
-```bash -# RL — train SAC on the Go2 -python -m luckylab.scripts.train go2_velocity_flat \ - --agent.algorithm sac --agent.backend skrl --device cuda - -# IL — train ACT on a local dataset -python -m luckylab.scripts.train piper_pickandplace \ - --il.policy act --il.dataset-repo-id piper/pickandplace --device cuda -``` +5. Follow the prompts to ensure: + - Action Gate is **Enabled** + - Server is **Running** + - Scene is **Playing** -### Evaluate + -```bash -# RL — with keyboard control -python -m luckylab.scripts.play go2_velocity_flat \ - --algorithm sac --checkpoint runs/go2_velocity_sac/checkpoints/best_agent.pt \ - --keyboard - -# IL -python -m luckylab.scripts.play piper_pickandplace \ - --policy act --checkpoint runs/luckylab_il/final -``` +gRPC Panel -Keyboard controls: **W/S** forward/back, **A/D** strafe, **Q/E** turn, **Space** zero, **Esc** quit. +
-### Visualize +### 3. Run Debug Viewer ```bash -# Browse a dataset in Rerun (opens in browser) -python -m luckylab.scripts.visualize_dataset \ - --repo-id piper/pickandplace --episode-index 0 --web - -# List all registered tasks -python -m luckylab.scripts.list_envs +# Run the gRPC viewer script for your OS +./run_debug_viewer.bat # Windows +./run_debug_viewer.sh # Linux ``` ---- - -## Reinforcement Learning - -Four algorithms across two backends, all configurable via CLI or Python: - -| Algorithm | Type | Backends | -|-----------|------|----------| -| **PPO** | On-policy | skrl, sb3 | -| **SAC** | Off-policy | skrl, sb3 | -| **TD3** | Off-policy | skrl, sb3 | -| **DDPG** | Off-policy | skrl, sb3 | +If everything has been configured correctly, this script will log the inputs/outputs between LuckyLab and LuckyEngine, and display the camera feed being exported from LuckyEngine to LuckyLab. +### 4. Download & Run Piper Block Stacking Demo Model ```bash -python -m luckylab.scripts.train go2_velocity_flat \ - --agent.algorithm sac --agent.backend skrl \ - --agent.max-iterations 5000 \ - --env.num-envs 4096 \ - --device cuda -``` - -```python -from luckylab.rl import train, RlRunnerCfg -from luckylab.tasks import load_env_cfg - -env_cfg = load_env_cfg("go2_velocity_flat") -rl_cfg = RlRunnerCfg(algorithm="sac", backend="skrl", max_iterations=5000) -train(env_cfg=env_cfg, rl_cfg=rl_cfg, device="cuda") -``` - -> **Note:** LuckyEngine does not currently support environment parallelization, so on-policy algorithms like PPO that depend on large batch collection are not recommended. Off-policy algorithms like SAC are the best fit for now. Parallelization support is actively being worked on. +# Run the model download script for your OS +# Windows +./download_demo.bat +./run_demo.bat -> **Backend recommendation:** Stable Baselines3 is not designed for GPU training. If you want to train on GPU, use the skrl backend (`--agent.backend skrl`). - ---- - -## Imitation Learning - -LuckyLab integrates with [LeRobot](https://github.com/huggingface/lerobot) for imitation learning. ACT and Diffusion Policy are ready to use out of the box. Other LeRobot policies (Pi0, SmolVLA, etc.) are supported but require registering a task config for them first, similar to how the ACT and Diffusion configs are set up. - -```bash -python -m luckylab.scripts.train piper_pickandplace \ - --il.policy act \ - --il.dataset-repo-id piper/pickandplace \ - --il.batch-size 8 \ - --il.num-train-steps 100000 \ - --device cuda +# Linux +./download_demo.sh +./run_demo.sh ``` -Datasets are loaded from the [HuggingFace Hub](https://huggingface.co/datasets) or from a local directory at `~/.luckyrobots/data/` (configurable via `LUCKYROBOTS_DATA_HOME`). +Manually downloaded models need to be placed within their own subfolder within the /runs/ directory of LuckyLab, where-as the download scripts already extract to the appropriate nested location. --- -## Tasks - -Tasks bundle an environment config with RL and/or IL configs. The registry makes it easy to add new ones: - -```python -from luckylab.tasks import register_task -from luckylab.envs import ManagerBasedRlEnvCfg -from luckylab.rl import RlRunnerCfg - -env_cfg = ManagerBasedRlEnvCfg( - decimation=4, - robot="unitreego2", - scene="velocity", - observations={...}, - actions={...}, - rewards={...}, - terminations={...}, -) - -register_task( - "my_task", - env_cfg, - rl_cfgs={"ppo": RlRunnerCfg(algorithm="ppo", max_iterations=3000)}, -) -``` - ---- - -## Architecture - -LuckyLab uses a manager-based environment where each MDP component is handled by a dedicated manager, configured with direct function references: - -``` -ManagerBasedRlEnv -├── ObservationManager Observation groups with noise, delay, and history -├── ActionManager Action scaling, offset, and joint commands -├── RewardManager Weighted sum of reward terms -├── TerminationManager Episode termination conditions -└── CurriculumManager Progressive difficulty adjustment -``` - -```python -from luckylab.managers import RewardTermCfg, TerminationTermCfg -from luckylab.tasks.velocity import mdp +## How It Works -rewards = { - "track_velocity": RewardTermCfg(func=mdp.track_linear_velocity, weight=2.0, params={"std": 0.5}), - "action_rate": RewardTermCfg(func=mdp.action_rate_l2, weight=-0.1), -} +```mermaid +graph TD + LE[LuckyEngine] -terminations = { - "time_out": TerminationTermCfg(func=mdp.time_out, time_out=True), - "fell_over": TerminationTermCfg(func=mdp.bad_orientation, params={"limit_angle": 1.2}), -} -``` + LE <--> LR[luckyrobots client] ---- + LR --> ENV -## Visualization & Logging + subgraph LuckyLab ["    LuckyLab    "] + ENV[ManagerBasedEnv] + ENV --- OBS[Observations] + ENV --- ACT[Actions] + ENV --- REW[Rewards] + ENV --- TERM[Terminations] + ENV --- CURR[Curriculum] + end -**Policy Viewer** — a web-based MuJoCo viewer powered by [Viser](https://github.com/nerfstudio-project/viser) for inspecting trained RL policies. Renders the robot in a browser with velocity command sliders, pause/play, speed control, and keyboard input — no LuckyEngine connection required. + subgraph Backends ["Training Backends"] + SKRL[skrl — RL] + SB3[SB3 — RL] + LEROBOT[LeRobot — IL] + end -```bash -# Open http://localhost:8080 after starting -python -m luckylab.viewer.run_policy runs/go2_velocity_sac/checkpoints/best_agent.pt -``` + ENV --> Backends -**Rerun** — live step-by-step inspection of observations, actions, rewards, and camera feeds. No LuckyEngine connection required. + style LE fill:#1a1a2e,stroke:#0984e3,stroke-width:2px,color:#74b9ff + style LR fill:#1a1a2e,stroke:#00b894,stroke-width:2px,color:#55efc4 -```bash -# Dataset viewer -python -m luckylab.scripts.visualize_dataset --repo-id piper/pickandplace --web + style LuckyLab fill:#16213e,stroke:#6c5ce7,stroke-width:2px,color:#a29bfe + style ENV fill:#1a1a2e,stroke:#6c5ce7,stroke-width:2px,color:#a29bfe + style OBS fill:#1a1a2e,stroke:#636e72,stroke-width:1px,color:#dfe6e9 + style ACT fill:#1a1a2e,stroke:#636e72,stroke-width:1px,color:#dfe6e9 + style REW fill:#1a1a2e,stroke:#636e72,stroke-width:1px,color:#dfe6e9 + style TERM fill:#1a1a2e,stroke:#636e72,stroke-width:1px,color:#dfe6e9 + style CURR fill:#1a1a2e,stroke:#636e72,stroke-width:1px,color:#dfe6e9 -# Attach to evaluation -python -m luckylab.scripts.play go2_velocity_flat --algorithm sac --checkpoint best_agent.pt --rerun + style Backends fill:#16213e,stroke:#e17055,stroke-width:2px,color:#fab1a0 + style SKRL fill:#1a1a2e,stroke:#e17055,stroke-width:2px,color:#fab1a0 + style SB3 fill:#1a1a2e,stroke:#e17055,stroke-width:2px,color:#fab1a0 + style LEROBOT fill:#1a1a2e,stroke:#fdcb6e,stroke-width:2px,color:#ffeaa7 ``` -**Weights & Biases** — enabled by default for both RL and IL. Disable with `--agent.wandb false` or `--il.wandb false`. +LuckyEngine handles all physics simulation (built on MuJoCo). LuckyLab is purely a training orchestrator — it does not run physics locally. The [luckyrobots](https://github.com/luckyrobots/luckyrobots) package manages the gRPC connection, engine lifecycle, and domain randomization protocol. --- -## Project Structure +## Status -``` -src/luckylab/ -├── configs/ Simulation contract and shared configs -├── entity/ Robot entity and observation data -├── envs/ ManagerBasedRlEnv and MDP functions -│ └── mdp/ Observations, actions, rewards, terminations, curriculum -├── il/ Imitation learning -│ └── lerobot/ LeRobot integration (trainer, wrapper) -├── managers/ Observation, action, reward, termination, curriculum managers -├── rl/ Reinforcement learning -│ ├── skrl/ skrl backend -│ ├── sb3/ Stable Baselines3 backend -│ ├── config.py RlRunnerCfg and algorithm configs -│ └── common.py Shared utilities -├── scene/ Scene management -├── scripts/ CLI entry points (train, play, list_envs, visualize_dataset) -├── tasks/ Task definitions and registry -│ ├── velocity/ Locomotion velocity tracking -│ └── pickandplace/ Manipulation (IL) -├── utils/ NaN guard, noise models, rerun logger, keyboard, buffers -└── viewer/ Debug visualization with Viser -``` +LuckyLab is in **early development (alpha)**. The Piper block-stacking demo above is the current focus. The codebase also includes scaffolding for reinforcement learning (Go2 velocity tracking via [skrl](https://github.com/Toni-SM/skrl) / [Stable Baselines3](https://github.com/DLR-RM/stable-baselines3)) and additional imitation learning policies via [LeRobot](https://github.com/huggingface/lerobot). --- ## Development ```bash +# Manual install with uv (instead of setup scripts) uv sync --all-groups uv run pre-commit install @@ -260,19 +143,3 @@ uv run pytest tests -v uv run ruff check src tests uv run ruff format src tests ``` - -See [CONTRIBUTING.md](CONTRIBUTING.md) for details. - ---- - -## Acknowledgments - -LuckyLab is inspired by: -- [MJLab](https://github.com/google-deepmind/mujoco_playground) — manager-based, config-driven environment architecture -- [LeRobot](https://github.com/huggingface/lerobot) — imitation learning policies and dataset format - -Built on top of [skrl](https://github.com/Toni-SM/skrl) and [Stable Baselines3](https://github.com/DLR-RM/stable-baselines3) for RL training. - -## License - -MIT License — see [LICENSE](LICENSE) for details. diff --git a/download_demo.bat b/download_demo.bat new file mode 100644 index 0000000..483a0eb --- /dev/null +++ b/download_demo.bat @@ -0,0 +1,39 @@ +@echo off +setlocal + +set REPO=luckyrobots/luckylab +set TAG=demo-v0.1.0 +set DEMO_NAME=piper_blockstacking_act +set ZIP_NAME=%DEMO_NAME%.zip +set DOWNLOAD_URL=https://github.com/%REPO%/releases/download/%TAG%/%ZIP_NAME% + +rem Resolve the directory this script lives in (the luckylab root) +rem %~dp0 has a trailing backslash — remove it so quoted paths don't break +set SCRIPT_DIR=%~dp0 +if "%SCRIPT_DIR:~-1%"=="\" set "SCRIPT_DIR=%SCRIPT_DIR:~0,-1%" + +echo Downloading demo from %DOWNLOAD_URL% ... +curl -L "%DOWNLOAD_URL%" -o "%SCRIPT_DIR%\%ZIP_NAME%" +if %errorlevel% neq 0 ( + echo ERROR: Download failed. Make sure curl is available and the URL is correct. + pause + exit /b 1 +) + +echo Extracting demo ... +powershell -Command "Expand-Archive -Path '%SCRIPT_DIR%\%ZIP_NAME%' -DestinationPath '%SCRIPT_DIR%' -Force" +if %errorlevel% neq 0 ( + echo ERROR: Extraction failed. + pause + exit /b 1 +) + +del "%SCRIPT_DIR%\%ZIP_NAME%" + +echo. +echo Demo installed successfully. +echo Model: runs\%DEMO_NAME%\final\ +echo Script: run_demo.bat +echo. +echo Run 'run_demo.bat' to start the demo. +pause diff --git a/download_demo.sh b/download_demo.sh new file mode 100644 index 0000000..e491b3a --- /dev/null +++ b/download_demo.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +set -euo pipefail + +REPO="luckyrobots/luckylab" +TAG="demo-v0.1.0" +DEMO_NAME="piper_blockstacking_act" +ZIP_NAME="${DEMO_NAME}.zip" +DOWNLOAD_URL="https://github.com/${REPO}/releases/download/${TAG}/${ZIP_NAME}" + +# Resolve the directory this script lives in (the luckylab root) +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +echo "Downloading demo from ${DOWNLOAD_URL} ..." +curl -L "${DOWNLOAD_URL}" -o "${SCRIPT_DIR}/${ZIP_NAME}" + +echo "Extracting demo ..." +unzip -o "${SCRIPT_DIR}/${ZIP_NAME}" -d "${SCRIPT_DIR}" + +chmod -R u+rwX "${SCRIPT_DIR}/runs" +chmod +x "${SCRIPT_DIR}/run_demo.sh" + +rm "${SCRIPT_DIR}/${ZIP_NAME}" + +echo "" +echo "Demo installed successfully." +echo " Model: runs/${DEMO_NAME}/final/" +echo " Script: run_demo.sh" +echo "" +echo "Run './run_demo.sh' to start the demo." diff --git a/grpc_debug_viewer.py b/grpc_debug_viewer.py new file mode 100644 index 0000000..bfc80dc --- /dev/null +++ b/grpc_debug_viewer.py @@ -0,0 +1,562 @@ +#!/usr/bin/env python3 +""" +gRPC Debug Viewer — live camera frames + metadata from LuckyEngine. + +Two modes: + 1. Active (default): Drives physics via Session.step() with zero actions. + Camera frames are returned synchronized with each physics step. + 2. Passive (--passive): Uses CameraService.StreamCamera for read-only + viewing without driving physics. Scene must already be playing. + +Usage (from GrpcPanel "Run Command"): + uv run --no-sync --group il python ../../../scripts/grpc_debug_viewer.py --cameras Camera --width 256 --height 256 + +Usage (standalone): + python scripts/grpc_debug_viewer.py --cameras Camera TopCamera --width 256 --height 256 + python scripts/grpc_debug_viewer.py --passive --cameras Camera --width 640 --height 480 --fps 30 + +Dependencies: pillow, numpy, grpcio, luckyrobots (all in the luckylab 'il' group) +""" + +from __future__ import annotations + +import argparse +import math +import signal +import sys +import threading +import time +import tkinter as tk +from collections import deque +from typing import Optional + +import numpy as np +from PIL import Image, ImageDraw, ImageFont, ImageTk + + +_shutdown = False + + +def _signal_handler(sig, frame): + global _shutdown + _shutdown = True + + +def _ts() -> str: + return time.strftime("%H:%M:%S") + + +def log(msg: str) -> None: + print(f"[{_ts()}] {msg}", flush=True) + + +def fmt_vec(values: list[float], max_items: int = 8) -> str: + if not values: + return "[]" + preview = ", ".join(f"{v:+.3f}" for v in values[:max_items]) + suffix = f", ... ({len(values)} total)" if len(values) > max_items else "" + return f"[{preview}{suffix}]" + + +def raw_to_rgb_image(data: bytes, width: int, height: int, channels: int) -> Image.Image: + """Convert raw pixel bytes to a PIL RGB Image.""" + arr = np.frombuffer(data, dtype=np.uint8) + + if channels == 4: + arr = arr.reshape((height, width, 4)) + return Image.fromarray(arr[:, :, :3], "RGB") + elif channels == 3: + arr = arr.reshape((height, width, 3)) + return Image.fromarray(arr, "RGB") + elif channels == 1: + arr = arr.reshape((height, width)) + return Image.fromarray(arr, "L").convert("RGB") + else: + arr = arr.reshape((height, width, channels)) + return Image.fromarray(arr[:, :, :3], "RGB") + + +class FpsCounter: + def __init__(self, window: int = 60): + self._times: deque[float] = deque(maxlen=window) + + def tick(self) -> float: + now = time.perf_counter() + self._times.append(now) + if len(self._times) < 2: + return 0.0 + elapsed = self._times[-1] - self._times[0] + return (len(self._times) - 1) / elapsed if elapsed > 0 else 0.0 + + +class BandwidthTracker: + """Tracks accumulated bytes sent and received.""" + + GRPC_OVERHEAD = 80 # approximate per-message framing/header bytes + + def __init__(self): + self.total_sent = 0 + self.total_received = 0 + + def record_step(self, action_count: int, obs: object) -> None: + # Sent: actions (float32) + camera requests + proto overhead + sent = action_count * 4 + self.GRPC_OVERHEAD + self.total_sent += sent + + # Received: observation floats + action floats + camera frames + overhead + recv = self.GRPC_OVERHEAD + if obs.observation: + recv += len(obs.observation) * 4 + if obs.actions: + recv += len(obs.actions) * 4 + for cf in obs.camera_frames: + recv += len(cf.data) + 32 # frame data + per-frame proto fields + self.total_received += recv + + def record_stream_frame(self, data_len: int) -> None: + """Record a passive-mode streamed camera frame.""" + self.total_received += data_len + 32 + self.GRPC_OVERHEAD + + @property + def total(self) -> int: + return self.total_sent + self.total_received + + @staticmethod + def fmt(nbytes: int) -> str: + if nbytes < 1024: + return f"{nbytes} B" + elif nbytes < 1024 * 1024: + return f"{nbytes / 1024:.1f} KB" + elif nbytes < 1024 * 1024 * 1024: + return f"{nbytes / (1024 * 1024):.1f} MB" + else: + return f"{nbytes / (1024 * 1024 * 1024):.2f} GB" + + +class ViewerWindow: + """Tkinter window that displays camera frames with an FPS overlay.""" + + def __init__(self, root: tk.Tk, title: str, width: int, height: int): + self.window = tk.Toplevel(root) + self.window.title(title) + self.window.protocol("WM_DELETE_WINDOW", self._on_close) + self.window.bind("", self._on_key) + + # Scale up small frames for visibility (minimum 256px on shortest side) + self._display_scale = max(1, 256 // min(width, height)) if min(width, height) < 256 else 1 + self._dw = width * self._display_scale + self._dh = height * self._display_scale + + self.canvas = tk.Canvas(self.window, width=self._dw, height=self._dh, bg="black") + self.canvas.pack() + + self._photo: Optional[ImageTk.PhotoImage] = None + self._image_id = self.canvas.create_image(0, 0, anchor=tk.NW) + self._closed = False + + # Force window to appear on top + self.window.lift() + self.window.attributes("-topmost", True) + self.window.after(500, lambda: self.window.attributes("-topmost", False)) + + def _on_close(self): + global _shutdown + _shutdown = True + self._closed = True + + def _on_key(self, event): + if event.char == "q": + global _shutdown + _shutdown = True + self._closed = True + + @property + def closed(self) -> bool: + return self._closed + + def update_frame(self, img: Image.Image, fps: float, frame_number: int, + bw: BandwidthTracker | None = None): + if self._closed: + return + + # Draw overlay + draw = ImageDraw.Draw(img) + line1 = f"FPS: {fps:.1f} Frame: {frame_number}" + overlay_h = 18 + if bw is not None: + line2 = f"Sent: {bw.fmt(bw.total_sent)} Recv: {bw.fmt(bw.total_received)} Total: {bw.fmt(bw.total)}" + overlay_h = 32 + draw.rectangle([(0, 0), (img.width, overlay_h)], fill=(0, 0, 0, 128)) + draw.text((4, 2), line1, fill=(0, 255, 0)) + if bw is not None: + draw.text((4, 16), line2, fill=(0, 255, 0)) + + # Scale up if needed + if self._display_scale > 1: + img = img.resize((self._dw, self._dh), Image.NEAREST) + + self._photo = ImageTk.PhotoImage(img) + self.canvas.itemconfig(self._image_id, image=self._photo) + self.canvas.update_idletasks() + + +# ── Active mode: drive physics with zero actions ────────────────────── + + +def run_active(args: argparse.Namespace, root: tk.Tk) -> int: + from luckyrobots import Session + + log(f"Active mode: connecting to {args.host}:{args.port} (robot={args.robot})") + + session = Session(host=args.host, port=args.port) + try: + session.connect(timeout_s=args.timeout, robot=args.robot) + except Exception as e: + log(f"Connection failed: {e}") + return 1 + + # Query schema + client = session.engine_client + schema_resp = client.get_agent_schema(agent_name=args.agent) + schema = schema_resp.schema + log(f"Agent: {schema.agent_name}") + log(f" observation_size: {schema.observation_size}") + log(f" action_size: {schema.action_size}") + if schema.observation_names: + log(f" observation_names: {list(schema.observation_names)[:12]}...") + if schema.action_names: + log(f" action_names: {list(schema.action_names)[:12]}...") + + action_size = schema.action_size or 12 + + # Configure cameras + cam_cfgs = [ + {"name": name, "width": args.width, "height": args.height} + for name in args.cameras + ] + session.configure_cameras(cam_cfgs) + log(f"Configured {len(cam_cfgs)} camera(s): {args.cameras} @ {args.width}x{args.height}") + + # Reset + log("Resetting agent...") + try: + obs = session.reset(agent_name=args.agent) + log(f"Reset OK. frame={obs.frame_number}, cameras={len(obs.camera_frames)}") + for cf in obs.camera_frames: + arr = np.frombuffer(cf.data, dtype=np.uint8) if cf.data else np.array([]) + log( + f" reset cam='{cf.name}' {cf.width}x{cf.height}x{cf.channels} " + f"bytes={len(cf.data)} " + f"min={arr.min() if len(arr) else 'N/A'} " + f"max={arr.max() if len(arr) else 'N/A'} " + f"mean={arr.mean():.1f}" if len(arr) else "EMPTY" + ) + except Exception as e: + log(f"Reset failed: {e}") + return 1 + + # Create viewer windows + windows: dict[str, ViewerWindow] = {} + for name in args.cameras: + windows[name] = ViewerWindow(root, f"Camera: {name}", args.width, args.height) + + fps_counter = FpsCounter() + bw_tracker = BandwidthTracker() + frame_idx = 0 + metadata_interval = max(1, int(args.metadata_hz)) + + log("Starting step loop. Press 'q' in any window or close window to quit.") + log("=" * 60) + + def step_loop(): + nonlocal frame_idx, obs + global _shutdown + + if _shutdown: + session.close(stop_engine=False) + log("Session closed.") + root.quit() + return + + # Generate actions: small sinusoidal wiggle or zeros + if args.wiggle: + t = time.perf_counter() + actions = [ + math.sin(t * 2.0 + i * 0.7) * args.wiggle_amp + for i in range(action_size) + ] + else: + actions = [0.0] * action_size + + try: + obs = session.step(actions=actions, agent_name=args.agent) + except Exception as e: + log(f"Step failed: {e}") + session.close(stop_engine=False) + root.quit() + return + + fps = fps_counter.tick() + frame_idx += 1 + bw_tracker.record_step(action_size, obs) + + # Update camera windows + if frame_idx <= 3 and not obs.camera_frames: + log(f" WARNING: step returned 0 camera frames (configured: {args.cameras})") + + for cf in obs.camera_frames: + # Diagnostic: log first few frames + if frame_idx <= 3: + arr = np.frombuffer(cf.data, dtype=np.uint8) if cf.data else np.array([]) + log( + f" cam='{cf.name}' {cf.width}x{cf.height}x{cf.channels} " + f"bytes={len(cf.data)} " + f"min={arr.min() if len(arr) else 'N/A'} " + f"max={arr.max() if len(arr) else 'N/A'} " + f"mean={arr.mean():.1f}" if len(arr) else "EMPTY" + ) + + if len(cf.data) == 0: + continue + win = windows.get(cf.name) + if win and not win.closed: + img = raw_to_rgb_image(cf.data, cf.width, cf.height, cf.channels) + win.update_frame(img, fps, obs.frame_number, bw_tracker) + + # Print metadata periodically + if frame_idx % metadata_interval == 0: + log( + f"frame={obs.frame_number:>6} " + f"ts={obs.timestamp_ms}ms " + f"fps={fps:>5.1f} " + f"obs={fmt_vec(obs.observation)} " + f"act={fmt_vec(obs.actions)} " + f"cams={len(obs.camera_frames)}" + ) + + # Schedule next step immediately (as fast as physics allows) + root.after(1, step_loop) + + # Kick off the loop + root.after(1, step_loop) + return 0 + + +# ── Passive mode: read-only camera stream ───────────────────────────── + + +def run_passive(args: argparse.Namespace, root: tk.Tk) -> int: + import grpc + from luckyrobots.grpc.generated import camera_pb2, camera_pb2_grpc + + target = f"{args.host}:{args.port}" + log(f"Passive mode: connecting to {target}") + + channel = grpc.insecure_channel(target) + stub = camera_pb2_grpc.CameraServiceStub(channel) + + # List cameras if none specified + camera_names = list(args.cameras) + if not camera_names: + try: + resp = stub.ListCameras(camera_pb2.ListCamerasRequest(), timeout=5.0) + camera_names = [c.name for c in resp.cameras] + log(f"Discovered {len(camera_names)} camera(s): {camera_names}") + except Exception as e: + log(f"ListCameras failed: {e}") + return 1 + + if not camera_names: + log("No cameras available.") + return 1 + + # Create viewer windows + windows: dict[str, ViewerWindow] = {} + for name in camera_names: + windows[name] = ViewerWindow(root, f"Camera: {name}", args.width, args.height) + + fps_counters = {name: FpsCounter() for name in camera_names} + frame_counts = {name: 0 for name in camera_names} + + # Shared state for frames from streaming threads + _frame_lock = threading.Lock() + _latest_frames: dict[str, tuple] = {} # name -> (data, width, height, channels, frame_number, timestamp_ms) + _stream_errors: dict[str, str] = {} + + def stream_worker(cam_name: str): + """Background thread: iterate the gRPC stream, stash latest frame.""" + try: + req = camera_pb2.StreamCameraRequest( + name=cam_name, + target_fps=args.fps, + width=args.width, + height=args.height, + format="raw", + ) + stream = stub.StreamCamera(req, timeout=None) + for frame in stream: + if _shutdown: + break + with _frame_lock: + _latest_frames[cam_name] = ( + bytes(frame.data), + frame.width, + frame.height, + frame.channels, + frame.frame_number, + frame.timestamp_ms, + ) + except Exception as e: + msg = str(e) + if "StatusCode.CANCELLED" not in msg and "Cancelled" not in msg: + with _frame_lock: + _stream_errors[cam_name] = msg + + # Start stream threads + threads = [] + for name in camera_names: + t = threading.Thread(target=stream_worker, args=(name,), daemon=True) + t.start() + threads.append(t) + log(f"Started stream for '{name}' @ {args.width}x{args.height} target_fps={args.fps}") + + bw_tracker = BandwidthTracker() + + log("Streaming. Press 'q' in any window or close window to quit.") + log("=" * 60) + + metadata_interval = max(1, int(args.metadata_hz)) + + def poll_frames(): + global _shutdown + + if _shutdown: + channel.close() + log("Channel closed.") + root.quit() + return + + with _frame_lock: + frames = dict(_latest_frames) + _latest_frames.clear() + errors = dict(_stream_errors) + _stream_errors.clear() + + for cam_name, err_msg in errors.items(): + log(f"Stream '{cam_name}' error: {err_msg}") + + for cam_name, (data, w, h, ch, fnum, ts_ms) in frames.items(): + if len(data) == 0: + continue + + fps = fps_counters[cam_name].tick() + frame_counts[cam_name] += 1 + bw_tracker.record_stream_frame(len(data)) + + # Diagnostic: log pixel stats for first few frames + if frame_counts[cam_name] <= 3: + arr = np.frombuffer(data, dtype=np.uint8) + log( + f" [{cam_name}] DIAG: {w}x{h}x{ch} " + f"bytes={len(data)} " + f"min={arr.min()} max={arr.max()} mean={arr.mean():.1f}" + ) + + win = windows.get(cam_name) + if win and not win.closed: + img = raw_to_rgb_image(data, w, h, ch) + win.update_frame(img, fps, fnum, bw_tracker) + + if frame_counts[cam_name] % metadata_interval == 0: + log( + f"[{cam_name}] frame={fnum:>6} " + f"ts={ts_ms}ms " + f"fps={fps:>5.1f} " + f"{w}x{h}x{ch} " + f"bytes={len(data)}" + ) + + root.after(5, poll_frames) + + root.after(5, poll_frames) + return 0 + + +# ── Entry point ─────────────────────────────────────────────────────── + + +def main() -> int: + signal.signal(signal.SIGINT, _signal_handler) + + ap = argparse.ArgumentParser( + description="gRPC Debug Viewer — live camera frames + metadata from LuckyEngine" + ) + ap.add_argument("--host", default="127.0.0.1", help="gRPC server host") + ap.add_argument("--port", type=int, default=50051, help="gRPC server port") + ap.add_argument("--timeout", type=float, default=30.0, help="Connection timeout (seconds)") + + ap.add_argument( + "--cameras", + nargs="+", + default=["Camera"], + help="Camera entity name(s) to stream (default: Camera)", + ) + ap.add_argument("--width", type=int, default=256, help="Requested frame width") + ap.add_argument("--height", type=int, default=256, help="Requested frame height") + + ap.add_argument("--robot", default="so100", help="Robot name for Session.connect()") + ap.add_argument("--agent", default="", help="Agent name (empty = default)") + + ap.add_argument( + "--passive", + action="store_true", + help="Passive mode: use CameraService.StreamCamera instead of driving physics", + ) + ap.add_argument("--fps", type=int, default=30, help="Target FPS for passive stream") + + ap.add_argument( + "--wiggle", + action="store_true", + help="Send small sinusoidal actions to make the robot move (useful for visual confirmation)", + ) + ap.add_argument( + "--wiggle-amp", + type=float, + default=0.3, + help="Amplitude of wiggle actions (default: 0.3)", + ) + + ap.add_argument( + "--metadata-hz", + type=float, + default=1.0, + help="How often to print metadata to terminal (prints every N frames, where N = this value; default 1 = every frame)", + ) + + args = ap.parse_args() + + # Create root Tk window — keep it visible as a small control window + root = tk.Tk() + root.title("gRPC Debug Viewer") + root.geometry("300x60") + root.bind("", lambda e: _signal_handler(None, None) if e.char == "q" else None) + tk.Label(root, text="Press 'q' to quit").pack(pady=10) + + if args.passive: + rc = run_passive(args, root) + else: + rc = run_active(args, root) + + if rc != 0: + return rc + + try: + root.mainloop() + except KeyboardInterrupt: + log("Interrupted (Ctrl+C)") + + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/grpc_multicam_test.py b/grpc_multicam_test.py new file mode 100644 index 0000000..f9fc922 --- /dev/null +++ b/grpc_multicam_test.py @@ -0,0 +1,315 @@ +#!/usr/bin/env python3 +""" +Multi-Camera Stress Test — validates that N concurrent StreamCamera streams +don't starve the main thread. + +Connects in passive mode, opens one StreamCamera per camera entity, +and displays all feeds side-by-side with per-camera FPS counters. + +The key metric: with the CameraFrameBroker fix, FPS should stay close to +the editor frame rate regardless of camera count. Without the fix, +adding a second camera tanks from ~20 FPS to <1 FPS. + +Usage: + python grpc_multicam_test.py # auto-discover all cameras + python grpc_multicam_test.py --cameras Camera TopCamera + python grpc_multicam_test.py --cameras Camera TopCamera --width 640 --height 480 + +Dependencies: pillow, numpy, grpcio, luckyrobots (all in the luckylab 'il' group) +""" + +from __future__ import annotations + +import argparse +import math +import signal +import sys +import threading +import time +import tkinter as tk +from collections import deque +from typing import Optional + +import numpy as np +from PIL import Image, ImageDraw, ImageTk + + +_shutdown = False + + +def _signal_handler(sig, frame): + global _shutdown + _shutdown = True + + +def _ts() -> str: + return time.strftime("%H:%M:%S") + + +def log(msg: str) -> None: + print(f"[{_ts()}] {msg}", flush=True) + + +def raw_to_rgb_image(data: bytes, width: int, height: int, channels: int) -> Image.Image: + arr = np.frombuffer(data, dtype=np.uint8) + if channels == 4: + arr = arr.reshape((height, width, 4)) + return Image.fromarray(arr[:, :, :3], "RGB") + elif channels == 3: + arr = arr.reshape((height, width, 3)) + return Image.fromarray(arr, "RGB") + elif channels == 1: + arr = arr.reshape((height, width)) + return Image.fromarray(arr, "L").convert("RGB") + else: + arr = arr.reshape((height, width, channels)) + return Image.fromarray(arr[:, :, :3], "RGB") + + +class FpsCounter: + def __init__(self, window: int = 60): + self._times: deque[float] = deque(maxlen=window) + + def tick(self) -> float: + now = time.perf_counter() + self._times.append(now) + if len(self._times) < 2: + return 0.0 + elapsed = self._times[-1] - self._times[0] + return (len(self._times) - 1) / elapsed if elapsed > 0 else 0.0 + + +def main() -> int: + signal.signal(signal.SIGINT, _signal_handler) + + ap = argparse.ArgumentParser(description="Multi-camera gRPC stress test") + ap.add_argument("--host", default="127.0.0.1", help="gRPC server host") + ap.add_argument("--port", type=int, default=50051, help="gRPC server port") + ap.add_argument("--cameras", nargs="+", default=[], help="Camera names (empty = auto-discover)") + ap.add_argument("--width", type=int, default=256, help="Requested frame width") + ap.add_argument("--height", type=int, default=256, help="Requested frame height") + ap.add_argument("--fps", type=int, default=30, help="Target FPS per stream") + ap.add_argument("--duration", type=int, default=0, help="Auto-quit after N seconds (0 = run forever)") + + ap.add_argument("--wiggle", action="store_true", help="Drive physics with sinusoidal actions (requires Session)") + ap.add_argument("--wiggle-amp", type=float, default=0.3, help="Wiggle amplitude (default: 0.3)") + ap.add_argument("--robot", default="so100", help="Robot name for Session.connect()") + ap.add_argument("--agent", default="", help="Agent name (empty = default)") + ap.add_argument("--timeout", type=float, default=30.0, help="Connection timeout (seconds)") + + args = ap.parse_args() + + import grpc + from luckyrobots.grpc.generated import camera_pb2, camera_pb2_grpc + + target = f"{args.host}:{args.port}" + log(f"Connecting to {target}") + + channel = grpc.insecure_channel(target) + stub = camera_pb2_grpc.CameraServiceStub(channel) + + # Discover cameras + camera_names = list(args.cameras) + if not camera_names: + try: + resp = stub.ListCameras(camera_pb2.ListCamerasRequest(), timeout=5.0) + camera_names = [c.name for c in resp.cameras] + log(f"Auto-discovered {len(camera_names)} camera(s): {camera_names}") + except Exception as e: + log(f"ListCameras failed: {e}") + return 1 + + if not camera_names: + log("No cameras found.") + return 1 + + # ── Optional wiggle: set up Session to drive physics ── + session = None + action_size = 12 + if args.wiggle: + from luckyrobots import Session + + log(f"Wiggle mode: connecting Session (robot={args.robot})") + session = Session(host=args.host, port=args.port) + try: + session.connect(timeout_s=args.timeout, robot=args.robot) + except Exception as e: + log(f"Session connect failed: {e}") + return 1 + + schema_resp = session.engine_client.get_agent_schema(agent_name=args.agent) + action_size = schema_resp.schema.action_size or 12 + log(f" action_size={action_size}") + + cam_cfgs = [{"name": name, "width": args.width, "height": args.height} for name in camera_names] + session.configure_cameras(cam_cfgs) + + session.reset(agent_name=args.agent) + log("Session reset OK, wiggle active.") + + mode = "wiggle + passive streams" if args.wiggle else "passive streams only" + log(f"Streaming {len(camera_names)} camera(s): {camera_names} @ {args.width}x{args.height} target_fps={args.fps} [{mode}]") + log("=" * 60) + + # ── Tkinter setup: one window with all cameras side by side ── + + root = tk.Tk() + root.title(f"Multi-Camera Test ({len(camera_names)} cameras)") + root.bind("", lambda e: _signal_handler(None, None) if e.char == "q" else None) + root.protocol("WM_DELETE_WINDOW", lambda: _signal_handler(None, None)) + + display_scale = max(1, 256 // min(args.width, args.height)) if min(args.width, args.height) < 256 else 1 + dw = args.width * display_scale + dh = args.height * display_scale + total_w = dw * len(camera_names) + + canvas = tk.Canvas(root, width=total_w, height=dh + 40, bg="black") + canvas.pack() + + # Per-camera image items on canvas + cam_image_ids = {} + cam_photos: dict[str, Optional[ImageTk.PhotoImage]] = {} + for i, name in enumerate(camera_names): + x = i * dw + cam_image_ids[name] = canvas.create_image(x, 0, anchor=tk.NW) + cam_photos[name] = None + # Label + canvas.create_text(x + dw // 2, dh + 10, text=name, fill="white", font=("Consolas", 10)) + + # Status bar + status_id = canvas.create_text(total_w // 2, dh + 30, text="Starting...", fill="lime", font=("Consolas", 9)) + + # ── Streaming threads ── + + fps_counters = {name: FpsCounter() for name in camera_names} + frame_counts = {name: 0 for name in camera_names} + + _frame_lock = threading.Lock() + _latest_frames: dict[str, tuple] = {} + _stream_errors: dict[str, str] = {} + + start_time = time.perf_counter() + + def stream_worker(cam_name: str): + try: + req = camera_pb2.StreamCameraRequest( + name=cam_name, + target_fps=args.fps, + width=args.width, + height=args.height, + format="raw", + ) + stream = stub.StreamCamera(req, timeout=None) + for frame in stream: + if _shutdown: + break + with _frame_lock: + _latest_frames[cam_name] = ( + bytes(frame.data), + frame.width, + frame.height, + frame.channels, + frame.frame_number, + frame.timestamp_ms, + ) + except Exception as e: + msg = str(e) + if "StatusCode.CANCELLED" not in msg and "Cancelled" not in msg: + with _frame_lock: + _stream_errors[cam_name] = msg + + for name in camera_names: + t = threading.Thread(target=stream_worker, args=(name,), daemon=True) + t.start() + + # ── Poll & render loop ── + + def poll_frames(): + global _shutdown + + if _shutdown: + if session: + session.close(stop_engine=False) + channel.close() + log("Done.") + root.quit() + return + + # Auto-quit after duration + if args.duration > 0 and (time.perf_counter() - start_time) > args.duration: + _shutdown = True + + # Drive physics with wiggle actions + if session: + t = time.perf_counter() + actions = [math.sin(t * 2.0 + i * 0.7) * args.wiggle_amp for i in range(action_size)] + try: + session.step(actions=actions, agent_name=args.agent) + except Exception as e: + log(f"Step failed: {e}") + _shutdown = True + + with _frame_lock: + frames = dict(_latest_frames) + _latest_frames.clear() + errors = dict(_stream_errors) + _stream_errors.clear() + + for cam_name, err_msg in errors.items(): + log(f"STREAM ERROR [{cam_name}]: {err_msg}") + + fps_parts = [] + for cam_name, (data, w, h, ch, fnum, ts_ms) in frames.items(): + if len(data) == 0: + continue + + fps = fps_counters[cam_name].tick() + frame_counts[cam_name] += 1 + fps_parts.append(f"{cam_name}={fps:.1f}") + + img = raw_to_rgb_image(data, w, h, ch) + + # FPS overlay on each camera + draw = ImageDraw.Draw(img) + draw.rectangle([(0, 0), (img.width, 16)], fill=(0, 0, 0, 180)) + draw.text((4, 1), f"{fps:.1f} fps #{fnum}", fill=(0, 255, 0)) + + if display_scale > 1: + img = img.resize((dw, dh), Image.NEAREST) + + cam_photos[cam_name] = ImageTk.PhotoImage(img) + canvas.itemconfig(cam_image_ids[cam_name], image=cam_photos[cam_name]) + + # Log periodically + if frame_counts[cam_name] % 60 == 1: + log(f"[{cam_name}] frame={fnum:>6} fps={fps:.1f} {w}x{h}x{ch}") + + # Update status bar + if fps_parts: + elapsed = time.perf_counter() - start_time + canvas.itemconfig(status_id, text=f"FPS: {' | '.join(fps_parts)} elapsed: {elapsed:.0f}s") + + canvas.update_idletasks() + root.after(5, poll_frames) + + root.after(100, poll_frames) + + try: + root.mainloop() + except KeyboardInterrupt: + log("Interrupted") + + # Print summary + log("=" * 60) + log("SUMMARY:") + for name in camera_names: + count = frame_counts[name] + elapsed = time.perf_counter() - start_time + avg_fps = count / elapsed if elapsed > 0 else 0 + log(f" {name}: {count} frames in {elapsed:.1f}s = {avg_fps:.1f} avg fps") + + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/pyproject.toml b/pyproject.toml index 520f400..fd7e231 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -23,7 +23,7 @@ classifiers = [ "Programming Language :: Python :: 3.12", ] dependencies = [ - "luckyrobots>=0.1.82", + "luckyrobots>=0.1.84", "gymnasium>=0.29.1", "numpy>=1.24.0", "prettytable>=3.10.0", @@ -75,6 +75,15 @@ rerun = [ "rerun-sdk>=0.23.0", ] +[[tool.uv.index]] +name = "pytorch-cu128" +url = "https://download.pytorch.org/whl/cu128" +explicit = true + +[tool.uv.sources] +torch = { index = "pytorch-cu128" } + + [tool.pytest.ini_options] testpaths = ["tests"] python_files = ["test_*.py"] diff --git a/run_debug_viewer.bat b/run_debug_viewer.bat new file mode 100644 index 0000000..804355c --- /dev/null +++ b/run_debug_viewer.bat @@ -0,0 +1,19 @@ +@echo off +setlocal + +rem Resolve the directory this script lives in (the luckylab root) +set SCRIPT_DIR=%~dp0 +if "%SCRIPT_DIR:~-1%"=="\" set "SCRIPT_DIR=%SCRIPT_DIR:~0,-1%" + +cd /d "%SCRIPT_DIR%" + +echo Starting gRPC debug viewer (wiggle mode) ... +uv run --no-sync --group il python grpc_debug_viewer.py --cameras Camera --width 256 --height 256 --wiggle + +if %errorlevel% neq 0 ( + echo. + echo ERROR: Debug viewer failed to run. Make sure you have installed luckylab with: + echo uv sync --group il + pause + exit /b 1 +) diff --git a/run_debug_viewer.sh b/run_debug_viewer.sh new file mode 100644 index 0000000..3d3c3a3 --- /dev/null +++ b/run_debug_viewer.sh @@ -0,0 +1,13 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Resolve the directory this script lives in (the luckylab root) +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +cd "${SCRIPT_DIR}" + +echo "Starting gRPC debug viewer (wiggle mode) ..." +uv run --no-sync --group il python grpc_debug_viewer.py \ + --cameras Camera \ + --width 256 \ + --height 256 \ + --wiggle diff --git a/run_multicam_test.bat b/run_multicam_test.bat new file mode 100644 index 0000000..c7f91ab --- /dev/null +++ b/run_multicam_test.bat @@ -0,0 +1,25 @@ +@echo off +setlocal + +rem Multi-camera stress test for gRPC camera streaming. +rem Auto-discovers all cameras in the scene and streams them simultaneously. +rem Usage: run_multicam_test.bat [host] + +set SCRIPT_DIR=%~dp0 +if "%SCRIPT_DIR:~-1%"=="\" set "SCRIPT_DIR=%SCRIPT_DIR:~0,-1%" + +cd /d "%SCRIPT_DIR%" + +set HOST=%1 +if "%HOST%"=="" set HOST=127.0.0.1 + +echo Starting multi-camera stress test (host=%HOST%) ... +uv run --no-sync --group il python grpc_multicam_test.py --width 256 --height 256 --wiggle --host %HOST% + +if %errorlevel% neq 0 ( + echo. + echo ERROR: Test failed to run. Make sure you have installed luckylab with: + echo uv sync --group il + pause + exit /b 1 +) diff --git a/run_multicam_test.sh b/run_multicam_test.sh new file mode 100644 index 0000000..0bb2072 --- /dev/null +++ b/run_multicam_test.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Multi-camera stress test for gRPC camera streaming. +# Auto-discovers all cameras in the scene and streams them simultaneously. +# Usage: ./run_multicam_test.sh [host] + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +cd "${SCRIPT_DIR}" + +HOST="${1:-127.0.0.1}" + +echo "Starting multi-camera stress test (host=${HOST}) ..." +uv run --no-sync --group il python grpc_multicam_test.py \ + --width 256 \ + --height 256 \ + --wiggle \ + --host "${HOST}" diff --git a/setup.bat b/setup.bat new file mode 100644 index 0000000..6eb856f --- /dev/null +++ b/setup.bat @@ -0,0 +1,36 @@ +@echo off +setlocal + +echo Checking for uv... +where uv >nul 2>&1 +if %errorlevel% neq 0 ( + echo uv not found. Installing uv... + powershell -ExecutionPolicy ByPass -NoProfile -Command "irm https://astral.sh/uv/install.ps1 | iex" + if %errorlevel% neq 0 ( + echo Failed to install uv. + exit /b 1 + ) + echo uv installed successfully. +) else ( + echo uv is already installed. +) + +echo. +echo Running uv sync --all-groups... +uv sync --all-groups +if %errorlevel% neq 0 ( + echo uv sync failed. + exit /b 1 +) + +echo. +echo Installing pre-commit hooks... +uv run pre-commit install +if %errorlevel% neq 0 ( + echo pre-commit install failed. + exit /b 1 +) + +echo. +echo Setup complete! +endlocal diff --git a/setup.sh b/setup.sh new file mode 100755 index 0000000..35c1488 --- /dev/null +++ b/setup.sh @@ -0,0 +1,22 @@ +#!/usr/bin/env bash +set -e + +echo "Checking for uv..." +if ! command -v uv &> /dev/null; then + echo "uv not found. Installing uv..." + curl -LsSf https://astral.sh/uv/install.sh | sh + echo "uv installed successfully." +else + echo "uv is already installed." +fi + +echo +echo "Running uv sync --all-groups..." +uv sync --all-groups + +echo +echo "Installing pre-commit hooks..." +uv run pre-commit install + +echo +echo "Setup complete!" diff --git a/src/luckylab/envs/manager_based_rl_env.py b/src/luckylab/envs/manager_based_rl_env.py index eef89a4..a97b656 100644 --- a/src/luckylab/envs/manager_based_rl_env.py +++ b/src/luckylab/envs/manager_based_rl_env.py @@ -77,9 +77,6 @@ class ManagerBasedRlEnvCfg: """Per-step physics timeout in seconds (default 30s; engine default is 5s).""" skip_launch: bool = True """Skip launching engine (connect to existing).""" - simulation_mode: str = "fast" - """Simulation timing mode: 'fast' (training), 'realtime' (visualization), 'deterministic' (reproducibility).""" - # MDP configs. rewards: dict[str, RewardTermCfg] = field(default_factory=dict) """Reward terms: name -> config.""" @@ -310,14 +307,6 @@ def close(self) -> None: self.luckyrobots.close(stop_engine=not self.cfg.skip_launch) print_info("Environment closed") - def set_simulation_mode(self, mode: str) -> None: - """Change simulation timing mode. - - Args: - mode: 'fast' (training), 'realtime' (visualization), 'deterministic' (reproducibility) - """ - self.luckyrobots.set_simulation_mode(mode) - @staticmethod def seed(seed: int) -> None: """Set the random seed. @@ -342,7 +331,6 @@ def _connect(self) -> None: task=self.cfg.task, timeout_s=self.cfg.timeout_s, ) - self.luckyrobots.set_simulation_mode(self.cfg.simulation_mode) # Override the default 5s per-RPC timeout on the underlying gRPC client. if self.luckyrobots.engine_client is not None: self.luckyrobots.engine_client.timeout = self.cfg.step_timeout_s diff --git a/src/luckylab/il/config.py b/src/luckylab/il/config.py index 82343f1..438ea96 100644 --- a/src/luckylab/il/config.py +++ b/src/luckylab/il/config.py @@ -80,5 +80,3 @@ class IlRunnerCfg: """Per-step physics timeout in seconds.""" skip_launch: bool = True """Skip launching engine (connect to existing).""" - simulation_mode: str = "realtime" - """Simulation timing mode. IL inference always uses 'realtime'.""" diff --git a/src/luckylab/il/lerobot/wrapper.py b/src/luckylab/il/lerobot/wrapper.py index f8d5455..481ac8d 100644 --- a/src/luckylab/il/lerobot/wrapper.py +++ b/src/luckylab/il/lerobot/wrapper.py @@ -129,9 +129,6 @@ def make_lerobot_env( timeout_s=il_cfg.timeout_s, ) - # IL inference runs in realtime — physics steps at real-world speed. - session.set_simulation_mode("realtime") - # Override the default 5s per-RPC timeout on the underlying gRPC client. if session.engine_client is not None: session.engine_client.timeout = il_cfg.step_timeout_s diff --git a/src/luckylab/rl/common.py b/src/luckylab/rl/common.py index 726e28a..1fa309d 100644 --- a/src/luckylab/rl/common.py +++ b/src/luckylab/rl/common.py @@ -50,7 +50,6 @@ def print_config( t.add_row(["Device", device]) t.add_row(["Timesteps", f"{rl_cfg.max_iterations:,}"]) t.add_row(["Seed", rl_cfg.seed]) - t.add_row(["Sim Mode", env_cfg.simulation_mode]) t.add_row(["Episode", f"{env_cfg.episode_length_s}s ({env.max_episode_length} steps @ {1/env.step_dt:.0f} Hz)"]) if rl_cfg.wandb: t.add_row(["Wandb", rl_cfg.wandb_project]) diff --git a/src/luckylab/rl/sb3/wrapper.py b/src/luckylab/rl/sb3/wrapper.py index 7d01443..8001d27 100644 --- a/src/luckylab/rl/sb3/wrapper.py +++ b/src/luckylab/rl/sb3/wrapper.py @@ -80,11 +80,6 @@ def _get_obs_dim(self, dim) -> int: return sum(d[0] if isinstance(d, tuple) else d for d in dim) return dim - @property - def is_realtime(self) -> bool: - """Check if running in realtime mode.""" - return getattr(self.cfg, "simulation_mode", "fast") == "realtime" - def _obs_to_numpy(self, obs_dict: dict[str, torch.Tensor]) -> np.ndarray: """Extract policy obs, squeeze batch dim, convert to numpy.""" obs = obs_dict["policy"] diff --git a/src/luckylab/rl/skrl/wrapper.py b/src/luckylab/rl/skrl/wrapper.py index f468896..8c63770 100644 --- a/src/luckylab/rl/skrl/wrapper.py +++ b/src/luckylab/rl/skrl/wrapper.py @@ -80,11 +80,6 @@ def class_name(cls) -> str: def unwrapped(self) -> ManagerBasedRlEnv: return self.env - @property - def is_realtime(self) -> bool: - """Check if running in realtime mode.""" - return getattr(self.cfg, "simulation_mode", "fast") == "realtime" - @property def episode_length_buf(self) -> torch.Tensor: return self.unwrapped.episode_length_buf diff --git a/src/luckylab/scripts/play.py b/src/luckylab/scripts/play.py index 63b85fc..34bd97e 100644 --- a/src/luckylab/scripts/play.py +++ b/src/luckylab/scripts/play.py @@ -16,6 +16,7 @@ import copy import sys +import time from dataclasses import dataclass import numpy as np @@ -85,8 +86,6 @@ def run_play_rl(task: str, cfg: PlayRlConfig) -> int: print_info(str(e), color="red") return 1 - env_cfg.simulation_mode = "realtime" - rl_cfg = load_rl_cfg(task, cfg.algorithm) if rl_cfg is None: rl_cfg = RlRunnerCfg(algorithm=cfg.algorithm) @@ -241,7 +240,6 @@ def run_play_il(task: str, cfg: PlayIlConfig) -> int: session.connect(timeout_s=il_cfg.timeout_s, robot=il_cfg.robot) print_info(f"Connected to LuckyEngine") - session.set_simulation_mode("realtime") if session.engine_client is not None: session.engine_client.timeout = il_cfg.step_timeout_s @@ -308,8 +306,32 @@ def run_play_il(task: str, cfg: PlayIlConfig) -> int: ) print_info(f"Running evaluation for {cfg.episodes} episodes (max {cfg.max_episode_steps} steps)...") + eval_start_time = time.perf_counter() episode_lengths = [] + # Generate a unique run ID for progress tracking + run_id = f"{task}_{cfg.policy}_{int(eval_start_time)}" + + # Helper to report progress to the engine (fire-and-forget) + def _report(phase: str, ep: int = 0, step: int = 0, status: str = "", finished: bool = False): + if not hasattr(session, "report_progress"): + return + session.report_progress( + run_id=run_id, + task_name=task, + policy_name=cfg.policy.upper(), + phase=phase, + current_episode=ep, + total_episodes=cfg.episodes, + current_step=step, + max_steps=cfg.max_episode_steps, + elapsed_s=time.perf_counter() - eval_start_time, + status_text=status, + finished=finished, + ) + + _report("loading", status="Loading policy and configuring environment") + # Number of settle steps to run after reset before policy takes over. # Matches the engine's RESET_SETTLE_STEPS (20 at 50Hz = 0.4s). SETTLE_STEPS = 25 @@ -326,6 +348,7 @@ def run_play_il(task: str, cfg: PlayIlConfig) -> int: obs, _, _, _, _ = env.step(zero_action) steps = 0 + _report("running", ep=ep + 1, step=0, status=f"Episode {ep + 1}/{cfg.episodes}: settling") for _ in range(cfg.max_episode_steps): # Convert obs to tensors with batch dim; preprocessor handles @@ -345,6 +368,11 @@ def run_play_il(task: str, cfg: PlayIlConfig) -> int: if rr_log is not None: rr_log.log_il_step(obs, action_np, step=steps) + # Report progress every 10 steps to avoid spamming + if steps % 10 == 0 or steps == 1: + _report("running", ep=ep + 1, step=steps, + status=f"Episode {ep + 1}/{cfg.episodes}: step {steps}/{cfg.max_episode_steps}") + sys.stdout.write(f"\r Episode {ep + 1}/{cfg.episodes} step={steps}/{cfg.max_episode_steps} ") sys.stdout.flush() @@ -352,12 +380,26 @@ def run_play_il(task: str, cfg: PlayIlConfig) -> int: break episode_lengths.append(steps) + _report("running", ep=ep + 1, step=steps, + status=f"Episode {ep + 1}/{cfg.episodes}: length={steps}") sys.stdout.write(f"\r Episode {ep + 1}/{cfg.episodes}: length={steps} \n") sys.stdout.flush() except KeyboardInterrupt: print_info("\nEvaluation interrupted by user", color="yellow") finally: + # Report completion BEFORE closing the session (which tears down gRPC) + total_time = time.perf_counter() - eval_start_time + if episode_lengths: + results_summary = ( + f"Episodes: {len(episode_lengths)} | " + f"Mean Length: {np.mean(episode_lengths):.1f} | " + f"Min: {np.min(episode_lengths)} | Max: {np.max(episode_lengths)} | " + f"Time: {total_time:.1f}s" + ) + _report("complete", ep=len(episode_lengths), step=0, + status=results_summary, + finished=True) if rr_log is not None: rr_log.close() env.close() @@ -375,6 +417,7 @@ def run_play_il(task: str, cfg: PlayIlConfig) -> int: print_info(f" Std Length: {np.std(episode_lengths):.1f}") print_info(f" Min Length: {np.min(episode_lengths)}") print_info(f" Max Length: {np.max(episode_lengths)}") + print_info(f" Total Time: {total_time:.1f}s") print_info("Evaluation complete!") return 0 diff --git a/src/luckylab/tasks/blockstacking/__init__.py b/src/luckylab/tasks/blockstacking/__init__.py index 6c57962..c967a0d 100644 --- a/src/luckylab/tasks/blockstacking/__init__.py +++ b/src/luckylab/tasks/blockstacking/__init__.py @@ -12,7 +12,7 @@ experiment_name="piper_blockstacking_act", robot="piper", scene="blockstacking", - host="172.24.160.1", + host="127.0.0.1", port=50051, ), "diffusion": IlRunnerCfg( @@ -21,7 +21,7 @@ experiment_name="piper_blockstacking_diffusion", robot="piper", scene="blockstacking", - host="172.24.160.1", + host="127.0.0.1", port=50051, ), },