diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index f09017991f..05460711ac 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -36,26 +36,32 @@ repos:
- id: check-yaml
- id: check-toml
- id: end-of-file-fixer
+ exclude: ^(outputs/|examples/hil_serl_simulation_training/outputs/)
- id: trailing-whitespace
+ exclude: ^(outputs/|examples/hil_serl_simulation_training/outputs/)
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.12.4
hooks:
- id: ruff-format
+ exclude: ^(outputs/|examples/hil_serl_simulation_training/outputs/)
- id: ruff
args: [--fix, --exit-non-zero-on-fix]
+ exclude: ^(outputs/|examples/hil_serl_simulation_training/outputs/)
- repo: https://github.com/adhtruong/mirrors-typos
rev: v1.34.0
hooks:
- id: typos
args: [--force-exclude]
+ exclude: ^(outputs/|examples/hil_serl_simulation_training/outputs/)
- repo: https://github.com/asottile/pyupgrade
rev: v3.20.0
hooks:
- id: pyupgrade
args: [--py310-plus]
+ exclude: ^(outputs/|examples/hil_serl_simulation_training/outputs/)
##### Markdown Quality #####
- repo: https://github.com/rbubley/mirrors-prettier
diff --git a/examples/grid_hil_serl/README.md b/examples/grid_hil_serl/README.md
new file mode 100644
index 0000000000..db9c00cfc8
--- /dev/null
+++ b/examples/grid_hil_serl/README.md
@@ -0,0 +1,105 @@
+# Grid HIL SERL Environment
+
+This example demonstrates a **simplified HIL-SERL setup** for computer vision-based grid position prediction. Instead of complex robotic manipulation, the algorithm learns to predict which of the 64 grid cells contains a red cube based on camera images, with human feedback during training. Episodes are single prediction attempts: if the guess is correct, the agent receives reward 1; otherwise reward 0.
+
+## Overview
+
+The environment consists of:
+- An 8x8 grid world with high-definition visual rendering
+- A red cube that randomly spawns at grid cell centers
+- Top-left origin coordinate system (0,0) = top-left corner
+- Automatic high-definition image capture (1920x1080)
+
+
+## Environment preview
+
+
+
+## Files
+
+- `grid_scene.xml` - Mujoco scene definition with 8x8 grid
+- `grid_cube_randomizer.py` - Main script for randomizing cube positions
+- `README.md` - This documentation
+
+## Usage
+
+### Install LeRobot (one time)
+Follow the main repository instructions (from repo root):
+```bash
+pip install -e ".[hilserl]"
+```
+
+### 2. Record Demonstrations (Optional - this repo already contains a recorded dataset)
+```bash
+# From the repository root
+python examples/grid_hil_serl/record_grid_demo.py \
+ --config_path examples/grid_hil_serl/record_grid_position_lerobot.json
+```
+
+### 3. Train HIL-SERL Policy
+```bash
+# Terminal 1: Start learner
+cd src
+python -m lerobot.scripts.rl.learner --config_path ../examples/grid_hil_serl/train_grid_position.json
+
+# Terminal 2: Start actor (with human feedback)
+cd src
+python -m lerobot.scripts.rl.actor --config_path ../examples/grid_hil_serl/train_grid_position.json
+```
+
+The actor prints a rolling accuracy over the last 50 episodes and saves a plot every
+10 episodes to `outputs/grid_position/accuracy_plots/` so you can monitor training
+progress without attaching a debugger.
+
+## Features
+
+This example gives you a fast, single‑step prediction task: every episode the
+cube appears in one of the 64 cells of an 8×8 grid and the policy must guess the
+cell from a high‑definition overhead image. Episodes are only one step long, so
+each prediction immediately becomes a labelled training example. Along the way the
+actor logs a rolling 50‑episode success rate and stores matplotlib accuracy plots,
+making it easy to gauge progress without additional tooling. Because the cubes are
+placed exactly at grid centres and the camera is fixed, the setup stays perfectly
+repeatable while still exercising the end‑to‑end vision→prediction loop.
+
+
+
+Accuracy typically climbs toward ~95 % after roughly 140 prediction episodes.
+
+## HIL-SERL Workflow
+
+This simplified setup demonstrates the core HIL-SERL concept with minimal complexity:
+
+### Training Phase (Offline)
+1. **Automatic Data Collection**: Environment randomly places cube in different grid positions
+2. **Supervised Learning**: Algorithm learns to predict grid position from images
+3. **Ground Truth Labels**: Exact grid coordinates provided for each image
+
+### Human-in-the-Loop Phase (Online)
+1. **Algorithm Prediction**: Model predicts cube position from camera images
+2. **Binary Feedback**: Human (or auto-supervision) accepts or corrects the guess
+3. **Iterative Learning**: Model improves based on the accepted/corrected outcome
+
+### Key Simplifications
+- **No Robot Control**: Focus purely on computer vision prediction
+- **Single-Step Episodes**: One prediction per episode with immediate success/failure reward
+- **Discrete Predictions**: 64 possible outputs (one per grid cell)
+- **Perfect Ground Truth**: Exact position labels available
+- **Visual Task Only**: No complex motor control or physics
+
+## Integration with LeRobot
+
+The environment integrates with LeRobot's HIL-SERL framework through:
+
+1. **Custom Gym Environment**: `GridPositionPrediction-v0` registered with gymnasium
+2. **LeRobot-Compatible Interface**: Proper observation/action space formatting
+3. **Config Files**: `record_grid_position.json` and `train_grid_position.json`
+4. **Dataset Collection**: Automated recording of image-position pairs
+
+## Dependencies
+
+- mujoco
+- numpy
+- PIL (Pillow)
+- gymnasium (optional, for integration)
+- matplotlib
diff --git a/examples/grid_hil_serl/grid_cube_randomizer.py b/examples/grid_hil_serl/grid_cube_randomizer.py
new file mode 100644
index 0000000000..45ee111f19
--- /dev/null
+++ b/examples/grid_hil_serl/grid_cube_randomizer.py
@@ -0,0 +1,161 @@
+#!/usr/bin/env python
+
+"""
+Random Grid Cube Spawner
+
+This script loads the 8x8 grid scene and randomly positions a cube
+in one of the 64 grid cells. The cube spawns at integer coordinates
+within the grid boundaries.
+"""
+
+import numpy as np
+import mujoco
+import mujoco.viewer
+import argparse
+import time
+from PIL import Image
+
+
+def save_camera_view(model, data, filename="img.jpg"):
+ """
+ Save the current camera view to a JPEG image file.
+
+ Args:
+ model: Mujoco model
+ data: Mujoco data
+ filename: Output filename (default: img.jpg)
+ """
+ try:
+ # Create a high-definition renderer for the current camera
+ renderer = mujoco.Renderer(model, height=1080, width=1920)
+
+ # Update the scene and render
+ renderer.update_scene(data, camera="grid_camera")
+ img = renderer.render()
+
+ if img is not None:
+ # Convert to PIL Image and save
+ image = Image.fromarray(img)
+ image.save(filename)
+ print(f"Camera view saved to: {filename}")
+ else:
+ print("Warning: Could not capture camera view")
+
+ # Clean up renderer (if close method exists)
+ if hasattr(renderer, 'close'):
+ renderer.close()
+
+ except Exception as e:
+ print(f"Error saving image: {e}")
+
+
+def randomize_cube_position(model, data, grid_size=8):
+ """
+ Randomly position the cube in one of the grid cells.
+
+ Args:
+ model: Mujoco model
+ data: Mujoco data
+ grid_size: Size of the grid (8x8)
+ """
+ # For 8x8 grid: generate random cell indices from 0-7 for both x and y
+ # This gives us coordinates for each of the 64 grid cells
+ x_cell = np.random.randint(0, 8) # 0 to 7 inclusive
+ y_cell = np.random.randint(0, 8) # 0 to 7 inclusive
+
+ # Convert cell indices to center positions (offset by 0.5 from grid lines)
+ # X: left(0) = -3.5, right(7) = 3.5
+ x_pos = (x_cell - grid_size // 2) + 0.5
+ # Y: top(0) = 3.5, bottom(7) = -3.5 (flipped coordinate system)
+ y_pos = (grid_size // 2 - y_cell) - 0.5
+
+ print(f"Spawning cube at grid cell ({x_cell}, {y_cell}) -> position ({x_pos}, {y_pos})")
+
+ # Set the cube position and velocity (free joint has 6 DOF: 3 pos + 3 vel)
+ cube_joint_id = mujoco.mj_name2id(model, mujoco.mjtObj.mjOBJ_JOINT, "cube_joint")
+
+ # Set position (x, y, z) - keep rotation as identity (0, 0, 0)
+ data.qpos[model.jnt_qposadr[cube_joint_id]:model.jnt_qposadr[cube_joint_id] + 6] = [x_pos, y_pos, 0.5, 0, 0, 0]
+
+ # Reset velocity to zero (linear and angular velocities)
+ data.qvel[model.jnt_dofadr[cube_joint_id]:model.jnt_dofadr[cube_joint_id] + 6] = [0, 0, 0, 0, 0, 0]
+
+ return x_pos, y_pos
+
+
+def run_grid_viewer(xml_path, randomize_interval=2.0, auto_save=True):
+ """
+ Run the grid viewer with random cube positioning.
+
+ Args:
+ xml_path: Path to the XML scene file
+ randomize_interval: How often to randomize cube position (seconds)
+ auto_save: Whether to automatically save camera view after each repositioning
+ """
+ print(f"Loading scene: {xml_path}")
+ model = mujoco.MjModel.from_xml_path(xml_path)
+ data = mujoco.MjData(model)
+
+ print("\n" + "="*50)
+ print("8x8 Grid Cube Randomizer")
+ print("="*50)
+ print("This scene shows an 8x8 grid with a randomly positioned cube.")
+ print(f"Cube position randomizes every {randomize_interval} seconds.")
+ print()
+ print("Controls:")
+ print(" R: Manually randomize cube position")
+ print(" S: Save current camera view to img.jpg")
+ print(" Space: Pause/unpause")
+ print(" Esc: Exit")
+ print(" Camera: Mouse controls for rotation/zoom")
+ print("="*50)
+
+ last_randomize_time = 0
+
+ with mujoco.viewer.launch_passive(model, data) as viewer:
+ # Initial randomization
+ x, y = randomize_cube_position(model, data)
+ mujoco.mj_forward(model, data)
+
+ while viewer.is_running():
+ current_time = time.time()
+
+ # Auto-randomize every few seconds
+ if current_time - last_randomize_time > randomize_interval:
+ x, y = randomize_cube_position(model, data)
+ mujoco.mj_forward(model, data)
+ # Force viewer to update the scene
+ viewer.sync()
+ # Save the current camera view if auto_save is enabled
+ if auto_save:
+ save_camera_view(model, data, "img.jpg")
+ last_randomize_time = current_time
+
+ # Small delay to prevent excessive CPU usage
+ time.sleep(0.01)
+
+ print("\nViewer closed.")
+
+
+def main():
+ parser = argparse.ArgumentParser(description="8x8 Grid Cube Randomizer")
+ parser.add_argument("--xml", type=str, default="grid_scene.xml",
+ help="Path to XML scene file")
+ parser.add_argument("--interval", type=float, default=3.0,
+ help="Randomization interval in seconds")
+ parser.add_argument("--no-save", action="store_true",
+ help="Disable automatic saving of camera views")
+
+ args = parser.parse_args()
+
+ try:
+ run_grid_viewer(args.xml, args.interval, not args.no_save)
+ except FileNotFoundError:
+ print(f"Error: Could not find XML file '{args.xml}'")
+ print("Make sure the XML file exists in the current directory.")
+ except Exception as e:
+ print(f"Error: {e}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/grid_hil_serl/grid_position_env.py b/examples/grid_hil_serl/grid_position_env.py
new file mode 100644
index 0000000000..2f30190ed5
--- /dev/null
+++ b/examples/grid_hil_serl/grid_position_env.py
@@ -0,0 +1,286 @@
+#!/usr/bin/env python
+
+"""
+Grid Position Prediction Environment for HIL-SERL
+
+This environment provides image observations and grid position labels for training
+a position prediction policy using HIL-SERL. Episodes are single-step: the agent
+predicts which grid cell contains the cube and immediately receives binary feedback.
+"""
+
+import numpy as np
+import gymnasium as gym
+from gymnasium import spaces
+import mujoco
+import mujoco.viewer
+from pathlib import Path
+from PIL import Image
+# Register the environment with gymnasium
+gym.register(
+ id="GridPositionPrediction-v0",
+ entry_point="examples.grid_hil_serl.grid_position_env:GridPositionLeRobotEnv",
+ kwargs={"xml_path": "grid_scene.xml"}
+)
+
+
+class GridPositionEnv(gym.Env):
+ """
+ Simplified environment for grid position prediction.
+
+ Observations: High-definition RGB images (1920x1080)
+ Actions: Discrete grid index (0-63)
+ Reward: Binary feedback from human (correct/incorrect prediction)
+ """
+
+ def __init__(self, xml_path="grid_scene.xml", render_mode="rgb_array", show_viewer: bool = True):
+ super().__init__()
+
+ # Load Mujoco model
+ self.xml_path = Path(__file__).parent / xml_path
+ if not self.xml_path.exists():
+ raise FileNotFoundError(f"XML file not found: {self.xml_path}")
+
+ self.model = mujoco.MjModel.from_xml_path(str(self.xml_path))
+ self.data = mujoco.MjData(self.model)
+
+ # Setup renderer for observations
+ self.renderer = mujoco.Renderer(self.model, height=1080, width=1920)
+ # Optional interactive viewer for visualization
+ self.viewer = None
+ self._show_viewer = bool(show_viewer)
+ if self._show_viewer:
+ try:
+ self.viewer = mujoco.viewer.launch_passive(self.model, self.data)
+ except Exception:
+ self.viewer = None
+
+ # Observation space: High-definition RGB images
+ self.observation_space = spaces.Box(
+ low=0,
+ high=255,
+ shape=(1080, 1920, 3),
+ dtype=np.uint8
+ )
+
+ self.grid_size = 8
+
+ # Action space: continuous pair in [-1, 1]
+ self.action_space = spaces.Box(
+ low=np.full(2, -1.0, dtype=np.float32),
+ high=np.full(2, 1.0, dtype=np.float32),
+ dtype=np.float32,
+ )
+
+ # Environment state
+ self.current_position = None
+
+
+ # Initialize to random position
+ self._randomize_cube_position()
+
+ def _randomize_cube_position(self):
+ """Randomize cube position and return grid coordinates."""
+ # Generate random cell indices (0-7 for 8x8 grid)
+ x_cell = np.random.randint(0, self.grid_size)
+ y_cell = np.random.randint(0, self.grid_size)
+
+ # Convert to physical positions (top-left origin)
+ x_pos = (x_cell - self.grid_size // 2) + 0.5
+ y_pos = (self.grid_size // 2 - y_cell) - 0.5
+
+ # Set cube position
+ cube_joint_id = mujoco.mj_name2id(self.model, mujoco.mjtObj.mjOBJ_JOINT, "cube_joint")
+ self.data.qpos[mujoco.mj_name2id(self.model, mujoco.mjtObj.mjOBJ_JOINT, "cube_joint"):mujoco.mj_name2id(self.model, mujoco.mjtObj.mjOBJ_JOINT, "cube_joint") + 6] = [x_pos, y_pos, 0.5, 0, 0, 0]
+ self.data.qvel[mujoco.mj_name2id(self.model, mujoco.mjtObj.mjOBJ_JOINT, "cube_joint"):mujoco.mj_name2id(self.model, mujoco.mjtObj.mjOBJ_JOINT, "cube_joint") + 6] = [0, 0, 0, 0, 0, 0]
+
+ mujoco.mj_forward(self.model, self.data)
+ if self.viewer is not None:
+ try:
+ self.viewer.sync()
+ except Exception:
+ pass
+
+ # Store current grid position (0-63)
+ self.current_position = y_cell * self.grid_size + x_cell
+
+ return self.current_position
+
+ def reset(self, seed=None, options=None):
+ """Reset environment to random cube position."""
+ super().reset(seed=seed)
+
+ # Randomize cube position
+ position = self._randomize_cube_position()
+
+ # Get observation
+ obs = self._get_observation()
+
+ return obs, {"grid_position": position}
+
+ def step(self, action):
+ """Advance to the next random position (single-step episodes)."""
+ position = self._randomize_cube_position()
+ obs = self._get_observation()
+
+ # Placeholder reward; the wrapper computes the binary reward.
+ reward = 0.0
+ terminated = True
+ truncated = False
+
+ info = {
+ "grid_position": position,
+ "x_cell": position % self.grid_size,
+ "y_cell": position // self.grid_size
+ }
+
+ return obs, reward, terminated, truncated, info
+
+ def _get_observation(self):
+ """Get current camera observation."""
+ self.renderer.update_scene(self.data, camera="grid_camera")
+ img = self.renderer.render()
+ return img
+
+ def render(self):
+ """Render current observation."""
+ return self._get_observation()
+
+ def close(self):
+ """Clean up resources."""
+ # Renderer doesn't have a close method in this version
+ try:
+ if self.viewer is not None:
+ self.viewer.close()
+ except Exception:
+ pass
+
+ def get_current_position(self):
+ """Get current grid position (0-63)."""
+ return self.current_position
+
+ def validate_prediction(self, predicted_position):
+ """
+ Validate a predicted position against current cube position.
+
+ Args:
+ predicted_position: Integer 0-63 representing predicted grid cell
+
+ Returns:
+ bool: True if prediction is correct
+ """
+ return predicted_position == self.current_position
+
+
+# LeRobot-compatible wrapper
+class GridPositionLeRobotEnv:
+ """
+ LeRobot-compatible wrapper for the grid position environment.
+ """
+
+ def __init__(self, xml_path="grid_scene.xml"):
+ self.env = GridPositionEnv(xml_path, show_viewer=True)
+
+ # LeRobot observation space format - use 'observation.image' as expected by SAC
+ self.observation_space = {
+ "observation.image": spaces.Box(
+ low=0, high=255, shape=(1080, 1920, 3), dtype=np.uint8
+ )
+ }
+
+ # Action is predicted grid cell coordinates in [-1, 1]
+ self.action_space = spaces.Box(
+ low=np.full(2, -1.0, dtype=np.float32),
+ high=np.full(2, 1.0, dtype=np.float32),
+ dtype=np.float32,
+ )
+
+ def reset(self, seed=None, options=None):
+ obs, info = self.env.reset(seed=seed, options=options)
+
+ # Convert to LeRobot format
+ lerobot_obs = {
+ "observation.image": obs
+ }
+
+ return lerobot_obs, info
+
+ def step(self, action):
+ # Compute reward based on predicted grid index
+ gt_position = int(self.env.get_current_position())
+ gt_x = gt_position % self.env.grid_size
+ gt_y = gt_position // self.env.grid_size
+
+ action_arr = np.asarray(action, dtype=np.float32).reshape(-1)
+ if action_arr.size >= 2:
+ # Policy emits values in [-1, 1]; rescale to [0, grid_size - 1]
+ scaled = (np.clip(action_arr[:2], -1.0, 1.0) + 1.0) * 0.5 * (self.env.grid_size - 1)
+ pred_x = int(np.clip(np.rint(scaled[0]), 0, self.env.grid_size - 1))
+ pred_y = int(np.clip(np.rint(scaled[1]), 0, self.env.grid_size - 1))
+ else:
+ pred_x = pred_y = 0
+
+ # Binary reward: 1 for exact match, 0 otherwise
+ correct = pred_x == gt_x and pred_y == gt_y
+ reward = 1.0 if correct else 0.0
+
+ # Advance environment to next random position
+ obs, _, _, _, info = self.env.step([0.0, 0.0])
+ terminated = True
+ truncated = False
+
+ # Convert to LeRobot format
+ lerobot_obs = {
+ "observation.image": obs
+ }
+
+ # Enrich info with prediction details
+ l1_error = abs(pred_x - gt_x) + abs(pred_y - gt_y)
+
+ info = {
+ **info,
+ "gt_x": gt_x,
+ "gt_y": gt_y,
+ "gt_index": gt_position,
+ "pred_x": pred_x,
+ "pred_y": pred_y,
+ "pred_index": pred_y * self.env.grid_size + pred_x,
+ "correct": correct,
+ "error.l1": float(l1_error),
+ "reward": reward,
+ }
+
+ return lerobot_obs, reward, terminated, truncated, info
+
+ def render(self):
+ return self.env.render()
+
+ def close(self):
+ return self.env.close()
+
+
+if __name__ == "__main__":
+ # Test the environment
+ env = GridPositionEnv()
+
+ print("Testing Grid Position Environment")
+ print("=" * 40)
+
+ obs, info = env.reset()
+ print(f"Initial position: {info['grid_position']}")
+ print(f"Observation shape: {obs.shape}")
+
+ # Test a few steps
+ for i in range(5):
+ obs, reward, terminated, truncated, info = env.step(0)
+ pos = info['grid_position']
+ x_cell = info['x_cell']
+ y_cell = info['y_cell']
+ print(f"Step {i+1}: Grid cell ({x_cell}, {y_cell}) = Position {pos}")
+
+ # Save sample image
+ img = Image.fromarray(obs)
+ img.save(f"sample_position_{pos}.jpg")
+ print(f" Saved image: sample_position_{pos}.jpg")
+
+ env.close()
+ print("Environment test completed!")
diff --git a/examples/grid_hil_serl/grid_scene.xml b/examples/grid_hil_serl/grid_scene.xml
new file mode 100644
index 0000000000..8340aa74d0
--- /dev/null
+++ b/examples/grid_hil_serl/grid_scene.xml
@@ -0,0 +1,37 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/examples/grid_hil_serl/hil_serl_training_guide_from_the_website.md b/examples/grid_hil_serl/hil_serl_training_guide_from_the_website.md
new file mode 100644
index 0000000000..0e8cf168be
--- /dev/null
+++ b/examples/grid_hil_serl/hil_serl_training_guide_from_the_website.md
@@ -0,0 +1,926 @@
+# HIL-SERL Real Robot Training Workflow Guide
+
+In this tutorial you will go through the full Human-in-the-Loop Sample-Efficient Reinforcement Learning (HIL-SERL) workflow using LeRobot. You will master training a policy with RL on a real robot in just a few hours.
+
+HIL-SERL is a sample-efficient reinforcement learning algorithm that combines human demonstrations with online learning and human interventions. The approach starts from a small set of human demonstrations, uses them to train a reward classifier, and then employs an actor-learner architecture where humans can intervene during policy execution to guide exploration and correct unsafe behaviors. In this tutorial, you'll use a gamepad to provide interventions and control the robot during the learning process.
+
+It combines three key ingredients:
+
+1. **Offline demonstrations & reward classifier:** a handful of human-teleop episodes plus a vision-based success detector give the policy a shaped starting point.
+
+2. **On-robot actor / learner loop with human interventions:** a distributed Soft Actor Critic (SAC) learner updates the policy while an actor explores on the physical robot; the human can jump in at any time to correct dangerous or unproductive behaviour.
+
+3. **Safety & efficiency tools:** joint/end-effector (EE) bounds, crop region of interest (ROI) preprocessing and WandB monitoring keep the data useful and the hardware safe.
+
+Together these elements let HIL-SERL reach near-perfect task success and faster cycle times than imitation-only baselines.
+
+
+
+
+
+
+ HIL-SERL workflow, Luo et al. 2024
+
+
+This guide provides step-by-step instructions for training a robot policy using LeRobot's HilSerl implementation to train on a real robot.
+
+## What do I need?
+
+- A gamepad (recommended) or keyboard to control the robot
+- A Nvidia GPU
+- A real robot with a follower and leader arm (optional if you use the keyboard or the gamepad)
+- A URDF file for the robot for the kinematics package (check `lerobot/model/kinematics.py`)
+
+## What kind of tasks can I train?
+
+One can use HIL-SERL to train on a variety of manipulation tasks. Some recommendations:
+
+- Start with a simple task to understand how the system works.
+ - Push cube to a goal region
+ - Pick and lift cube with the gripper
+- Avoid extremely long horizon tasks. Focus on tasks that can be completed in 5-10 seconds.
+- Once you have a good idea of how the system works, you can try more complex tasks and longer horizons.
+ - Pick and place cube
+ - Bimanual tasks to pick objects with two arms
+ - Hand-over tasks to transfer objects from one arm to another
+ - Go crazy!
+
+## Install LeRobot with HIL-SERL
+
+To install LeRobot with HIL-SERL, you need to install the `hilserl` extra.
+
+```bash
+pip install -e ".[hilserl]"
+```
+
+## Real Robot Training Workflow
+
+### Understanding Configuration
+
+The training process begins with proper configuration for the HILSerl environment. The main configuration class is `GymManipulatorConfig` in `lerobot/rl/gym_manipulator.py`, which contains nested `HILSerlRobotEnvConfig` and `DatasetConfig`. The configuration is organized into focused, nested sub-configs:
+
+
+```python
+class GymManipulatorConfig:
+ env: HILSerlRobotEnvConfig # Environment configuration (nested)
+ dataset: DatasetConfig # Dataset recording/replay configuration (nested)
+ mode: str | None = None # "record", "replay", or None (for training)
+ device: str = "cpu" # Compute device
+
+class HILSerlRobotEnvConfig(EnvConfig):
+ robot: RobotConfig | None = None # Main robot agent (defined in `lerobot/robots`)
+ teleop: TeleoperatorConfig | None = None # Teleoperator agent, e.g., gamepad or leader arm
+ processor: HILSerlProcessorConfig # Processing pipeline configuration (nested)
+ name: str = "real_robot" # Environment name
+ task: str | None = None # Task identifier
+ fps: int = 10 # Control frequency
+
+# Nested processor configuration
+class HILSerlProcessorConfig:
+ control_mode: str = "gamepad" # Control mode
+ observation: ObservationConfig | None = None # Observation processing settings
+ image_preprocessing: ImagePreprocessingConfig | None = None # Image crop/resize settings
+ gripper: GripperConfig | None = None # Gripper control and penalty settings
+ reset: ResetConfig | None = None # Environment reset and timing settings
+ inverse_kinematics: InverseKinematicsConfig | None = None # IK processing settings
+ reward_classifier: RewardClassifierConfig | None = None # Reward classifier settings
+ max_gripper_pos: float | None = 100.0 # Maximum gripper position
+
+# Sub-configuration classes
+class ObservationConfig:
+ add_joint_velocity_to_observation: bool = False # Add joint velocities to state
+ add_current_to_observation: bool = False # Add motor currents to state
+ display_cameras: bool = False # Display camera feeds during execution
+
+class ImagePreprocessingConfig:
+ crop_params_dict: dict[str, tuple[int, int, int, int]] | None = None # Image cropping parameters
+ resize_size: tuple[int, int] | None = None # Target image size
+
+class GripperConfig:
+ use_gripper: bool = True # Enable gripper control
+ gripper_penalty: float = 0.0 # Penalty for inappropriate gripper usage
+
+class ResetConfig:
+ fixed_reset_joint_positions: Any | None = None # Joint positions for reset
+ reset_time_s: float = 5.0 # Time to wait during reset
+ control_time_s: float = 20.0 # Maximum episode duration
+ terminate_on_success: bool = True # Whether to terminate episodes on success detection
+
+class InverseKinematicsConfig:
+ urdf_path: str | None = None # Path to robot URDF file
+ target_frame_name: str | None = None # End-effector frame name
+ end_effector_bounds: dict[str, list[float]] | None = None # EE workspace bounds
+ end_effector_step_sizes: dict[str, float] | None = None # EE step sizes per axis
+
+class RewardClassifierConfig:
+ pretrained_path: str | None = None # Path to pretrained reward classifier
+ success_threshold: float = 0.5 # Success detection threshold
+ success_reward: float = 1.0 # Reward value for successful episodes
+
+# Dataset configuration
+class DatasetConfig:
+ repo_id: str # LeRobot dataset repository ID
+ task: str # Task identifier
+ root: str | None = None # Local dataset root directory
+ num_episodes_to_record: int = 5 # Number of episodes for recording
+ replay_episode: int | None = None # Episode index for replay
+ push_to_hub: bool = False # Whether to push datasets to Hub
+```
+
+
+### Processor Pipeline Architecture
+
+HIL-SERL uses a modular processor pipeline architecture that processes robot observations and actions through a series of composable steps. The pipeline is divided into two main components:
+
+#### Environment Processor Pipeline
+
+The environment processor (`env_processor`) handles incoming observations and environment state:
+
+1. **VanillaObservationProcessorStep**: Converts raw robot observations into standardized format
+2. **JointVelocityProcessorStep** (optional): Adds joint velocity information to observations
+3. **MotorCurrentProcessorStep** (optional): Adds motor current readings to observations
+4. **ForwardKinematicsJointsToEE** (optional): Computes end-effector pose from joint positions
+5. **ImageCropResizeProcessorStep** (optional): Crops and resizes camera images
+6. **TimeLimitProcessorStep** (optional): Enforces episode time limits
+7. **GripperPenaltyProcessorStep** (optional): Applies penalties for inappropriate gripper usage
+8. **RewardClassifierProcessorStep** (optional): Automated reward detection using vision models
+9. **AddBatchDimensionProcessorStep**: Converts data to batch format for neural network processing
+10. **DeviceProcessorStep**: Moves data to the specified compute device (CPU/GPU)
+
+#### Action Processor Pipeline
+
+The action processor (`action_processor`) handles outgoing actions and human interventions:
+
+1. **AddTeleopActionAsComplimentaryDataStep**: Captures teleoperator actions for logging
+2. **AddTeleopEventsAsInfoStep**: Records intervention events and episode control signals
+3. **InterventionActionProcessorStep**: Handles human interventions and episode termination
+4. **Inverse Kinematics Pipeline** (when enabled):
+ - **MapDeltaActionToRobotActionStep**: Converts delta actions to robot action format
+ - **EEReferenceAndDelta**: Computes end-effector reference and delta movements
+ - **EEBoundsAndSafety**: Enforces workspace safety bounds
+ - **InverseKinematicsEEToJoints**: Converts end-effector actions to joint targets
+ - **GripperVelocityToJoint**: Handles gripper control commands
+
+#### Configuration Examples
+
+**Basic Observation Processing**:
+
+```json
+{
+ "env": {
+ "processor": {
+ "observation": {
+ "add_joint_velocity_to_observation": true,
+ "add_current_to_observation": false,
+ "display_cameras": false
+ }
+ }
+ }
+}
+```
+
+**Image Processing**:
+
+```json
+{
+ "env": {
+ "processor": {
+ "image_preprocessing": {
+ "crop_params_dict": {
+ "observation.images.front": [180, 250, 120, 150],
+ "observation.images.side": [180, 207, 180, 200]
+ },
+ "resize_size": [128, 128]
+ }
+ }
+ }
+}
+```
+
+**Inverse Kinematics Setup**:
+
+```json
+{
+ "env": {
+ "processor": {
+ "inverse_kinematics": {
+ "urdf_path": "path/to/robot.urdf",
+ "target_frame_name": "end_effector",
+ "end_effector_bounds": {
+ "min": [0.16, -0.08, 0.03],
+ "max": [0.24, 0.2, 0.1]
+ },
+ "end_effector_step_sizes": {
+ "x": 0.02,
+ "y": 0.02,
+ "z": 0.02
+ }
+ }
+ }
+ }
+}
+```
+
+### Advanced Observation Processing
+
+The HIL-SERL framework supports additional observation processing features that can improve policy learning:
+
+#### Joint Velocity Processing
+
+Enable joint velocity estimation to provide the policy with motion information:
+
+```json
+{
+ "env": {
+ "processor": {
+ "observation": {
+ "add_joint_velocity_to_observation": true
+ }
+ }
+ }
+}
+```
+
+This processor:
+
+- Estimates joint velocities using finite differences between consecutive joint position readings
+- Adds velocity information to the observation state vector
+- Useful for policies that need motion awareness for dynamic tasks
+
+#### Motor Current Processing
+
+Monitor motor currents to detect contact forces and load conditions:
+
+```json
+{
+ "env": {
+ "processor": {
+ "observation": {
+ "add_current_to_observation": true
+ }
+ }
+ }
+}
+```
+
+This processor:
+
+- Reads motor current values from the robot's control system
+- Adds current measurements to the observation state vector
+- Helps detect contact events, object weights, and mechanical resistance
+- Useful for contact-rich manipulation tasks
+
+#### Combined Observation Processing
+
+You can enable multiple observation processing features simultaneously:
+
+```json
+{
+ "env": {
+ "processor": {
+ "observation": {
+ "add_joint_velocity_to_observation": true,
+ "add_current_to_observation": true,
+ "display_cameras": false
+ }
+ }
+ }
+}
+```
+
+**Note**: Enabling additional observation features increases the state space dimensionality, which may require adjusting your policy network architecture and potentially collecting more training data.
+
+### Finding Robot Workspace Bounds
+
+Before collecting demonstrations, you need to determine the appropriate operational bounds for your robot.
+
+This helps simplify the problem of learning on the real robot in two ways: 1) by limiting the robot's operational space to a specific region that solves the task and avoids unnecessary or unsafe exploration, and 2) by allowing training in end-effector space rather than joint space. Empirically, learning in joint space for reinforcement learning in manipulation is often a harder problem - some tasks are nearly impossible to learn in joint space but become learnable when the action space is transformed to end-effector coordinates.
+
+**Using lerobot-find-joint-limits**
+
+This script helps you find the safe operational bounds for your robot's end-effector. Given that you have a follower and leader arm, you can use the script to find the bounds for the follower arm that will be applied during training.
+Bounding the action space will reduce the redundant exploration of the agent and guarantees safety.
+
+```bash
+lerobot-find-joint-limits \
+ --robot.type=so100_follower \
+ --robot.port=/dev/tty.usbmodem58760431541 \
+ --robot.id=black \
+ --teleop.type=so100_leader \
+ --teleop.port=/dev/tty.usbmodem58760431551 \
+ --teleop.id=blue
+```
+
+**Workflow**
+
+1. Run the script and move the robot through the space that solves the task
+2. The script will record the minimum and maximum end-effector positions and the joint angles and prints them to the console, for example:
+ ```
+ Max ee position [0.2417 0.2012 0.1027]
+ Min ee position [0.1663 -0.0823 0.0336]
+ Max joint positions [-20.0, -20.0, -20.0, -20.0, -20.0, -20.0]
+ Min joint positions [50.0, 50.0, 50.0, 50.0, 50.0, 50.0]
+ ```
+3. Use these values in the configuration of your teleoperation device (TeleoperatorConfig) under the `end_effector_bounds` field
+
+**Example Configuration**
+
+```json
+"end_effector_bounds": {
+ "max": [0.24, 0.20, 0.10],
+ "min": [0.16, -0.08, 0.03]
+}
+```
+
+### Collecting Demonstrations
+
+With the bounds defined, you can safely collect demonstrations for training. Training RL with off-policy algorithm allows us to use offline datasets collected in order to improve the efficiency of the learning process.
+
+**Setting Up Record Mode**
+
+Create a configuration file for recording demonstrations (or edit an existing one like [env_config.json](https://huggingface.co/datasets/lerobot/config_examples/resolve/main/rl/env_config.json)):
+
+1. Set `mode` to `"record"` at the root level
+2. Specify a unique `repo_id` for your dataset in the `dataset` section (e.g., "username/task_name")
+3. Set `num_episodes_to_record` in the `dataset` section to the number of demonstrations you want to collect
+4. Set `env.processor.image_preprocessing.crop_params_dict` to `{}` initially (we'll determine crops later)
+5. Configure `env.robot`, `env.teleop`, and other hardware settings in the `env` section
+
+Example configuration section:
+
+```json
+{
+ "env": {
+ "type": "gym_manipulator",
+ "name": "real_robot",
+ "fps": 10,
+ "processor": {
+ "control_mode": "gamepad",
+ "observation": {
+ "display_cameras": false
+ },
+ "image_preprocessing": {
+ "crop_params_dict": {},
+ "resize_size": [128, 128]
+ },
+ "gripper": {
+ "use_gripper": true,
+ "gripper_penalty": 0.0
+ },
+ "reset": {
+ "reset_time_s": 5.0,
+ "control_time_s": 20.0
+ }
+ },
+ "robot": {
+ // ... robot configuration ...
+ },
+ "teleop": {
+ // ... teleoperator configuration ...
+ }
+ },
+ "dataset": {
+ "repo_id": "username/pick_lift_cube",
+ "root": null,
+ "task": "pick_and_lift",
+ "num_episodes_to_record": 15,
+ "replay_episode": 0,
+ "push_to_hub": true
+ },
+ "mode": "record",
+ "device": "cpu"
+}
+```
+
+### Using a Teleoperation Device
+
+Along with your robot, you will need a teleoperation device to control it in order to collect datasets of your task and perform interventions during the online training.
+We support using a gamepad or a keyboard or the leader arm of the robot.
+
+HIL-Serl learns actions in the end-effector space of the robot. Therefore, the teleoperation will control the end-effector's x,y,z displacements.
+
+For that we need to define a version of the robot that takes actions in the end-effector space. Check the robot class `SO100FollowerEndEffector` and its configuration `SO100FollowerEndEffectorConfig` for the default parameters related to the end-effector space.
+
+
+```python
+class SO100FollowerEndEffectorConfig(SO100FollowerConfig):
+ """Configuration for the SO100FollowerEndEffector robot."""
+
+ # Default bounds for the end-effector position (in meters)
+ end_effector_bounds: dict[str, list[float]] = field( # bounds for the end-effector in x,y,z direction
+ default_factory=lambda: {
+ "min": [-1.0, -1.0, -1.0], # min x, y, z
+ "max": [1.0, 1.0, 1.0], # max x, y, z
+ }
+ )
+
+ max_gripper_pos: float = 50 # maximum gripper position that the gripper will be open at
+
+ end_effector_step_sizes: dict[str, float] = field( # maximum step size for the end-effector in x,y,z direction
+ default_factory=lambda: {
+ "x": 0.02,
+ "y": 0.02,
+ "z": 0.02,
+ }
+ )
+```
+
+
+The `Teleoperator` defines the teleoperation device. You can check the list of available teleoperators in `lerobot/teleoperators`.
+
+**Setting up the Gamepad**
+
+The gamepad provides a very convenient way to control the robot and the episode state.
+
+To setup the gamepad, you need to set the `control_mode` to `"gamepad"` and define the `teleop` section in the configuration file.
+
+```json
+{
+ "env": {
+ "teleop": {
+ "type": "gamepad",
+ "use_gripper": true
+ },
+ "processor": {
+ "control_mode": "gamepad",
+ "gripper": {
+ "use_gripper": true
+ }
+ }
+ }
+}
+```
+
+
+
+
+
+ Gamepad button mapping for robot control and episode management
+
+
+**Setting up the SO101 leader**
+
+The SO101 leader arm has reduced gears that allows it to move and track the follower arm during exploration. Therefore, taking over is much smoother than the gearless SO100.
+
+To setup the SO101 leader, you need to set the `control_mode` to `"leader"` and define the `teleop` section in the configuration file.
+
+```json
+{
+ "env": {
+ "teleop": {
+ "type": "so101_leader",
+ "port": "/dev/tty.usbmodem585A0077921",
+ "use_degrees": true
+ },
+ "processor": {
+ "control_mode": "leader",
+ "gripper": {
+ "use_gripper": true
+ }
+ }
+ }
+}
+```
+
+In order to annotate the success/failure of the episode, **you will need** to use a keyboard to press `s` for success, `esc` for failure.
+During the online training, press `space` to take over the policy and `space` again to give the control back to the policy.
+
+
+Video: SO101 leader teleoperation
+
+
+
+
+
+
SO101 leader teleoperation example, the leader tracks the follower, press `space` to intervene
+
+
+**Recording Demonstrations**
+
+Start the recording process, an example of the config file can be found [here](https://huggingface.co/datasets/aractingi/lerobot-example-config-files/blob/main/env_config_so100.json):
+
+```bash
+python -m lerobot.rl.gym_manipulator --config_path src/lerobot/configs/env_config_so100.json
+```
+
+During recording:
+
+1. The robot will reset to the initial position defined in the configuration file `env.processor.reset.fixed_reset_joint_positions`
+2. Complete the task successfully
+3. The episode ends with a reward of 1 when you press the "success" button
+4. If the time limit is reached, or the fail button is pressed, the episode ends with a reward of 0
+5. You can rerecord an episode by pressing the "rerecord" button
+6. The process automatically continues to the next episode
+7. After recording all episodes, the dataset is pushed to the Hugging Face Hub (optional) and saved locally
+
+### Processing the Dataset
+
+After collecting demonstrations, process them to determine optimal camera crops.
+Reinforcement learning is sensitive to background distractions, so it is important to crop the images to the relevant workspace area.
+
+Visual RL algorithms learn directly from pixel inputs, making them vulnerable to irrelevant visual information. Background elements like changing lighting, shadows, people moving, or objects outside the workspace can confuse the learning process. Good ROI selection should:
+
+- Include only the essential workspace where the task happens
+- Capture the robot's end-effector and all objects involved in the task
+- Exclude unnecessary background elements and distractions
+
+Note: If you already know the crop parameters, you can skip this step and just set the `crop_params_dict` in the configuration file during recording.
+
+**Determining Crop Parameters**
+
+Use the `crop_dataset_roi.py` script to interactively select regions of interest in your camera images:
+
+```bash
+python -m lerobot.rl.crop_dataset_roi --repo-id username/pick_lift_cube
+```
+
+1. For each camera view, the script will display the first frame
+2. Draw a rectangle around the relevant workspace area
+3. Press 'c' to confirm the selection
+4. Repeat for all camera views
+5. The script outputs cropping parameters and creates a new cropped dataset
+
+Example output:
+
+```
+Selected Rectangular Regions of Interest (top, left, height, width):
+observation.images.side: [180, 207, 180, 200]
+observation.images.front: [180, 250, 120, 150]
+```
+
+
+
+
+
+
+ Interactive cropping tool for selecting regions of interest
+
+
+**Updating Configuration**
+
+Add these crop parameters to your training configuration:
+
+```json
+{
+ "env": {
+ "processor": {
+ "image_preprocessing": {
+ "crop_params_dict": {
+ "observation.images.side": [180, 207, 180, 200],
+ "observation.images.front": [180, 250, 120, 150]
+ },
+ "resize_size": [128, 128]
+ }
+ }
+ }
+}
+```
+
+**Recommended image resolution**
+
+Most vision-based policies have been validated on square inputs of either **128×128** (default) or **64×64** pixels. We therefore advise setting the resize_size parameter to [128, 128] – or [64, 64] if you need to save GPU memory and bandwidth. Other resolutions are possible but have not been extensively tested.
+
+### Training a Reward Classifier
+
+The reward classifier plays an important role in the HIL-SERL workflow by automating reward assignment and automatically detecting episode success. Instead of manually defining reward functions or relying on human feedback for every timestep, the reward classifier learns to predict success/failure from visual observations. This enables the RL algorithm to learn efficiently by providing consistent and automated reward signals based on the robot's camera inputs.
+
+This guide explains how to train a reward classifier for human-in-the-loop reinforcement learning implementation of LeRobot. Reward classifiers learn to predict the reward value given a state which can be used in an RL setup to train a policy.
+
+**Note**: Training a reward classifier is optional. You can start the first round of RL experiments by annotating the success manually with your gamepad or keyboard device.
+
+The reward classifier implementation in `modeling_classifier.py` uses a pretrained vision model to process the images. It can output either a single value for binary rewards to predict success/fail cases or multiple values for multi-class settings.
+
+**Collecting a Dataset for the reward classifier**
+
+Before training, you need to collect a dataset with labeled examples. The `record_dataset` function in `gym_manipulator.py` enables the process of collecting a dataset of observations, actions, and rewards.
+
+To collect a dataset, you need to modify some parameters in the environment configuration based on HILSerlRobotEnvConfig.
+
+```bash
+python -m lerobot.rl.gym_manipulator --config_path src/lerobot/configs/reward_classifier_train_config.json
+```
+
+**Key Parameters for Data Collection**
+
+- **mode**: set it to `"record"` to collect a dataset (at root level)
+- **dataset.repo_id**: `"hf_username/dataset_name"`, name of the dataset and repo on the hub
+- **dataset.num_episodes_to_record**: Number of episodes to record
+- **env.processor.reset.terminate_on_success**: Whether to automatically terminate episodes when success is detected (default: `true`)
+- **env.fps**: Number of frames per second to record
+- **dataset.push_to_hub**: Whether to push the dataset to the hub
+
+The `env.processor.reset.terminate_on_success` parameter allows you to control episode termination behavior. When set to `false`, episodes will continue even after success is detected, allowing you to collect more positive examples with the reward=1 label. This is crucial for training reward classifiers as it provides more success state examples in your dataset. When set to `true` (default), episodes terminate immediately upon success detection.
+
+**Important**: For reward classifier training, set `terminate_on_success: false` to collect sufficient positive examples. For regular HIL-SERL training, keep it as `true` to enable automatic episode termination when the task is completed successfully.
+
+Example configuration section for data collection:
+
+```json
+{
+ "env": {
+ "type": "gym_manipulator",
+ "name": "real_robot",
+ "fps": 10,
+ "processor": {
+ "reset": {
+ "reset_time_s": 5.0,
+ "control_time_s": 20.0,
+ "terminate_on_success": false
+ },
+ "gripper": {
+ "use_gripper": true
+ }
+ },
+ "robot": {
+ // ... robot configuration ...
+ },
+ "teleop": {
+ // ... teleoperator configuration ...
+ }
+ },
+ "dataset": {
+ "repo_id": "hf_username/dataset_name",
+ "dataset_root": "data/your_dataset",
+ "task": "reward_classifier_task",
+ "num_episodes_to_record": 20,
+ "replay_episode": null,
+ "push_to_hub": true
+ },
+ "mode": "record",
+ "device": "cpu"
+}
+```
+
+**Reward Classifier Configuration**
+
+The reward classifier is configured using `configuration_classifier.py`. Here are the key parameters:
+
+- **model_name**: Base model architecture (e.g., we mainly use `"helper2424/resnet10"`)
+- **model_type**: `"cnn"` or `"transformer"`
+- **num_cameras**: Number of camera inputs
+- **num_classes**: Number of output classes (typically 2 for binary success/failure)
+- **hidden_dim**: Size of hidden representation
+- **dropout_rate**: Regularization parameter
+- **learning_rate**: Learning rate for optimizer
+
+Example configuration for training the [reward classifier](https://huggingface.co/datasets/aractingi/lerobot-example-config-files/blob/main/reward_classifier_train_config.json):
+
+```json
+{
+ "policy": {
+ "type": "reward_classifier",
+ "model_name": "helper2424/resnet10",
+ "model_type": "cnn",
+ "num_cameras": 2,
+ "num_classes": 2,
+ "hidden_dim": 256,
+ "dropout_rate": 0.1,
+ "learning_rate": 1e-4,
+ "device": "cuda",
+ "use_amp": true,
+ "input_features": {
+ "observation.images.front": {
+ "type": "VISUAL",
+ "shape": [3, 128, 128]
+ },
+ "observation.images.side": {
+ "type": "VISUAL",
+ "shape": [3, 128, 128]
+ }
+ }
+ }
+}
+```
+
+**Training the Classifier**
+
+To train the classifier, use the `train.py` script with your configuration:
+
+```bash
+lerobot-train --config_path path/to/reward_classifier_train_config.json
+```
+
+**Deploying and Testing the Model**
+
+To use your trained reward classifier, configure the `HILSerlRobotEnvConfig` to use your model:
+
+
+```python
+config = GymManipulatorConfig(
+ env=HILSerlRobotEnvConfig(
+ processor=HILSerlProcessorConfig(
+ reward_classifier=RewardClassifierConfig(
+ pretrained_path="path_to_your_pretrained_trained_model"
+ )
+ ),
+ # Other environment parameters
+ ),
+ dataset=DatasetConfig(...),
+ mode=None # For training
+)
+```
+
+
+or set the argument in the json config file.
+
+```json
+{
+ "env": {
+ "processor": {
+ "reward_classifier": {
+ "pretrained_path": "path_to_your_pretrained_model",
+ "success_threshold": 0.7,
+ "success_reward": 1.0
+ },
+ "reset": {
+ "terminate_on_success": true
+ }
+ }
+ }
+}
+```
+
+Run `gym_manipulator.py` to test the model.
+
+```bash
+python -m lerobot.rl.gym_manipulator --config_path path/to/env_config.json
+```
+
+The reward classifier will automatically provide rewards based on the visual input from the robot's cameras.
+
+**Example Workflow for training the reward classifier**
+
+1. **Create the configuration files**:
+ Create the necessary json configuration files for the reward classifier and the environment. Check the examples [here](https://huggingface.co/datasets/lerobot/config_examples/resolve/main/reward_classifier/config.json).
+
+2. **Collect a dataset**:
+
+ ```bash
+ python -m lerobot.rl.gym_manipulator --config_path src/lerobot/configs/env_config.json
+ ```
+
+3. **Train the classifier**:
+
+ ```bash
+ lerobot-train --config_path src/lerobot/configs/reward_classifier_train_config.json
+ ```
+
+4. **Test the classifier**:
+ ```bash
+ python -m lerobot.rl.gym_manipulator --config_path src/lerobot/configs/env_config.json
+ ```
+
+### Training with Actor-Learner
+
+The LeRobot system uses a distributed actor-learner architecture for training. This architecture decouples robot interactions from the learning process, allowing them to run concurrently without blocking each other. The actor server handles robot observations and actions, sending interaction data to the learner server. The learner server performs gradient descent and periodically updates the actor's policy weights. You will need to start two processes: a learner and an actor.
+
+**Configuration Setup**
+
+Create a training configuration file (example available [here](https://huggingface.co/datasets/lerobot/config_examples/resolve/main/rl/train_config.json)). The training config is based on the main `TrainRLServerPipelineConfig` class in `lerobot/configs/train.py`.
+
+1. Configure the policy settings (`type="sac"`, `device`, etc.)
+2. Set `dataset` to your cropped dataset
+3. Configure environment settings with crop parameters
+4. Check the other parameters related to SAC in [configuration_sac.py](https://github.com/huggingface/lerobot/blob/main/src/lerobot/policies/sac/configuration_sac.py#L79).
+5. Verify that the `policy` config is correct with the right `input_features` and `output_features` for your task.
+
+**Starting the Learner**
+
+First, start the learner server process:
+
+```bash
+python -m lerobot.rl.learner --config_path src/lerobot/configs/train_config_hilserl_so100.json
+```
+
+The learner:
+
+- Initializes the policy network
+- Prepares replay buffers
+- Opens a `gRPC` server to communicate with actors
+- Processes transitions and updates the policy
+
+**Starting the Actor**
+
+In a separate terminal, start the actor process with the same configuration:
+
+```bash
+python -m lerobot.rl.actor --config_path src/lerobot/configs/train_config_hilserl_so100.json
+```
+
+The actor:
+
+- Connects to the learner via `gRPC`
+- Initializes the environment
+- Execute rollouts of the policy to collect experience
+- Sends transitions to the learner
+- Receives updated policy parameters
+
+**Training Flow**
+
+The training proceeds automatically:
+
+1. The actor executes the policy in the environment
+2. Transitions are collected and sent to the learner
+3. The learner updates the policy based on these transitions
+4. Updated policy parameters are sent back to the actor
+5. The process continues until the specified step limit is reached
+
+**Human in the Loop**
+
+- The key to learning efficiently is to have human interventions to provide corrective feedback and completing the task to aide the policy learning and exploration.
+- To perform human interventions, you can press the upper right trigger button on the gamepad (or the `space` key on the keyboard). This will pause the policy actions and allow you to take over.
+- A successful experiment is one where the human has to intervene at the start but then reduces the amount of interventions as the policy improves. You can monitor the intervention rate in the `wandb` dashboard.
+
+
+
+
+
+
+
+ Example showing how human interventions help guide policy learning over time
+
+
+
+- The figure shows the plot of the episodic reward over interaction step. The figure shows the effect of human interventions on the policy learning.
+- The orange curve is an experiment without any human interventions. While the pink and blue curves are experiments with human interventions.
+- We can observe that the number of steps where the policy starts achieving the maximum reward is cut by a quarter when human interventions are present.
+
+**Monitoring and Debugging**
+
+If you have `wandb.enable` set to `true` in your configuration, you can monitor training progress in real-time through the [Weights & Biases](https://wandb.ai/site/) dashboard.
+
+### Guide to Human Interventions
+
+The learning process is very sensitive to the intervention strategy. It will takes a few runs to understand how to intervene effectively. Some tips and hints:
+
+- Allow the policy to explore for a few episodes at the start of training.
+- Avoid intervening for long periods of time. Try to intervene in situation to correct the robot's behaviour when it goes off track.
+- Once the policy starts achieving the task, even if its not perfect, you can limit your interventions to simple quick actions like a simple grasping commands.
+
+The ideal behaviour is that your intervention rate should drop gradually during training as shown in the figure below.
+
+
+
+
+
+
+
+ Plot of the intervention rate during a training run on a pick and lift cube
+ task
+
+
+
+
+
+
+