Skip to content

Generate augmented scenes and add nre scene support #21

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
sameerchavan0027 wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Generate augmented scenes and add nre scene support #21

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 55 additions & 13 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
<h1 align="center"><span>MobilityGen</span></h1>

<div align="center">
A toolset built on <a href="https://developer.nvidia.com/isaac/sim">NVIDIA Isaac Sim</a> that
A toolset built on <a href="https://developer.nvidia.com/isaac/sim">NVIDIA Isaac Sim</a> that
allows you to easily collect data for mobile robots.
<br></br>
<div>
Expand All @@ -15,7 +15,7 @@ Read below to learn more.

## Overview

MobilityGen is a toolset built on [NVIDIA Isaac Sim](https://developer.nvidia.com/isaac/sim) that enables you to easily generate and collect data for mobile robots.
MobilityGen is a toolset built on [NVIDIA Isaac Sim](https://developer.nvidia.com/isaac/sim) that enables you to easily generate and collect data for mobile robots.

It supports

Expand All @@ -29,7 +29,7 @@ It supports
- Depth Images
- *If you're interested in more, [let us know!](https://github.com/NVlabs/MobilityGen/issues)*

- ***Many robot types***
- ***Many robot types***

- Differential drive - Jetbot, Carter
- Quadruped - Spot
Expand All @@ -56,6 +56,7 @@ To get started with MobilityGen follow the setup and usage instructions below!
- [How to implement a custom scenario](#how-to-custom-scenario)
- [📝 Data Format](#-data-format)
- [👏 Contributing](#-contributing)
-

<a id="setup"></a>
## 🛠️ Setup
Expand Down Expand Up @@ -98,7 +99,7 @@ Next, we'll call ``link_app.sh`` to link the Isaac Sim installation directory to
> This step is helpful as it (1) Enables us to use VS code autocompletion (2) Allows us to call ./app/python.sh to launch Isaac Sim Python scripts (3) Allows us to call ./app/isaac-sim.sh to launch Isaac Sim.
</details>

### Step 4 - Install other python dependencies (including C++ path planner) (for procedural generation)
### Step 4 - Install other python dependencies (including C++ path planner) (for procedural generation)

1. Install miscellaneous python dependencies

Expand All @@ -118,8 +119,6 @@ Next, we'll call ``link_app.sh`` to link the Isaac Sim installation directory to
../app/python.sh -m pip install -e .
```

> Note: If you run into an error related to pybind11 while running this command, you may try ``../app/python.sh -m pip install wheel`` and/or ``../app/python.sh -m pip install pybind11[global]``.

### Step 4 - Launch Isaac Sim

1. Navigate to the repo root
Expand All @@ -134,7 +133,7 @@ Next, we'll call ``link_app.sh`` to link the Isaac Sim installation directory to
./scripts/launch_sim.sh
```

That's it! If everything worked, you should see Isaac Sim open with a window titled ``MobilityGen`` appear.
That's it! If everything worked, you should see Isaac Sim open with a window titled ``MobilityGen`` appear.

<img src="./assets/extension_gui.png" height="640px">

Expand Down Expand Up @@ -225,7 +224,7 @@ Rendering the sensor data is done offline. To do this call the following
python scripts/replay_directory.py --render_interval=200
```

> Note: For speed for this tutorial, we use a render interval of 200. If our physics timestep is 200 FPS, this means we
> Note: For speed for this tutorial, we use a render interval of 200. If our physics timestep is 200 FPS, this means we
> render 1 image per second.

That's it! Now the data with renderings should be stored in ``~/MobilityGenData/replays``
Expand Down Expand Up @@ -306,7 +305,7 @@ After a few seconds, you should see the scene and occupancy map appear.

1. Click ``Start Recording`` to start recording data

2. Go grab some coffee!
2. Go grab some coffee!

> The procedural generated methods automatically determine when to reset (ie: if the robot collides with
> an object and needs to respawn). If you run into any issues with the procedural methods getting stuck, please let us know.
Expand Down Expand Up @@ -356,7 +355,7 @@ for generating random motions.

## 📝 Data Format

MobilityGen records two types of data.
MobilityGen records two types of data.

- *Static Data* is recorded at the beginning of a recording
- Occupancy map
Expand Down Expand Up @@ -399,8 +398,6 @@ The state_dict has the following schema
"robot.front_camera.left.depth_image": np.ndarray, # [HxW], np.fp32 - Depth in meters
"robot.front_camera.left.segmentation_image": np.ndarray, # [HxW], np.uint8 - Segmentation class index
"robot.front_camera.left.segmentation_info": dict, # see Isaac replicator segmentation info format
"robot.front_camera.left.position": np.ndarray, # [3] - XYZ camera world position
"robot.front_camera.left.orientation": np.ndarray, # [4] - Quaternion camera world orientation
...
}
```
Expand Down Expand Up @@ -447,12 +444,57 @@ In case you're interested, each recording is represented as a directory with the
Most of the state information is captured under the ``state/common`` folder, as dictionary in a single ``.npy`` file.

However, for some data (images) this is inefficient. These instead get captured in their own folder based on the data
type and the name. (ie: rgb/robot.front_camera.left.depth_image).
type and the name. (ie: rgb/robot.front_camera.left.depth_image).

The name of each file corresponds to its physics timestep.

If you have any questions regarding the data logged by MobilityGen, please [let us know!](https://github.com/NVlabs/MobilityGen/issues)

### Augmented Scene Generation

The `scripts/generate_augmented_scenes.py` script allows you to generate augmented scenes with randomly placed obstacles. This is useful for creating training data or testing scenarios.

### Usage

To generate augmented scenes, run:
```bash
python scripts/generate_augmented_scenes.py \
--env_url="http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Simple_Warehouse/warehouse_multiple_shelves.usd" \
--output_dir="~/MobilityGenData/augmented_scenes" \
--num_scenes=1 \
--num_obstacles=10
```

#### Arguments:
- `--env_url`: Path or URL to the environment USD file
- `--output_dir`: Directory where generated scenes will be saved
- `--num_scenes`: Number of scenes to generate (default: 1)
- `--num_obstacles`: Number of obstacles per scene (default: 20)

### Output Structure
```
output_dir/
├── base_omap/ # Base occupancy map without obstacles
│ └── occupancy_map/ # ROS-format occupancy map
│ ├── map.yaml # Map configuration
│ └── map.png # Map image
├── scene_000/ # First generated scene
│ ├── stage.usd # Scene with obstacles
│ └── occupancy_map/ # ROS-format occupancy map
│ ├── map.yaml # Map configuration
│ └── map.png # Map image
├── scene_001/ # Second generated scene
│ ├── stage.usd # Scene with obstacles
│ └── occupancy_map/ # ROS-format occupancy map
│ ├── map.yaml # Map configuration
│ └── map.png # Map image
└── ... # Additional scenes
```

Each scene contains:
- A USD stage file with the environment and placed obstacles
- An occupancy map showing free and occupied space
- No additional subfolders or categorization

## 👏 Contributing
This [Developer Certificate of Origin](https://developercertificate.org/) applies to this project.
Expand Down
24 changes: 15 additions & 9 deletions exts/omni.ext.mobility_gen/omni/ext/mobility_gen/build.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,23 +24,24 @@
from omni.ext.mobility_gen.occupancy_map import OccupancyMap
from omni.ext.mobility_gen.config import Config
from omni.ext.mobility_gen.utils.occupancy_map_utils import occupancy_map_generate_from_prim_async
from omni.ext.mobility_gen.utils.global_utils import new_stage, new_world, set_viewport_camera
from omni.ext.mobility_gen.utils.global_utils import new_stage, new_world, set_viewport_camera, get_stage
from omni.ext.mobility_gen.scenarios import Scenario, SCENARIOS
from omni.ext.mobility_gen.robots import ROBOTS
from omni.ext.mobility_gen.reader import Reader



def load_scenario(path: str) -> Scenario:
reader = Reader(path)
config = reader.read_config()
robot_type = ROBOTS.get(config.robot_type)
scenario_type = SCENARIOS.get(config.scenario_type)
open_stage(os.path.join(path, "stage.usd"))
prim_utils.delete_prim("/World/robot")
stage = get_stage()
prim_path = str(stage.GetDefaultPrim().GetPath())
prim_utils.delete_prim(os.path.join(prim_path, "robot"))
new_world(physics_dt=robot_type.physics_dt)
occupancy_map = reader.read_occupancy_map()
robot = robot_type.build("/World/robot")
robot = robot_type.build(os.path.join(prim_path, "robot"))
chase_camera_path = robot.build_chase_camera()
set_viewport_camera(chase_camera_path)
robot_type = ROBOTS.get(config.robot_type)
Expand All @@ -54,14 +55,19 @@ def load_scenario(path: str) -> Scenario:
async def build_scenario_from_config(config: Config):
robot_type = ROBOTS.get(config.robot_type)
scenario_type = SCENARIOS.get(config.scenario_type)
new_stage()

open_stage(config.scene_usd)
stage = get_stage()
print("Default prim path: ", str(stage.GetDefaultPrim().GetPath()))
prim_path = str(stage.GetDefaultPrim().GetPath())

world = new_world(physics_dt=robot_type.physics_dt)
await world.initialize_simulation_context_async()
add_reference_to_stage(config.scene_usd,"/World/scene")
objects.GroundPlane("/World/ground_plane", visible=False)
robot = robot_type.build("/World/robot")

objects.GroundPlane(os.path.join(prim_path, "ground_plane"), visible=False)
robot = robot_type.build(os.path.join(prim_path,"robot"))
occupancy_map = await occupancy_map_generate_from_prim_async(
"/World/scene",
prim_path,
cell_size=robot.occupancy_map_cell_size,
z_min=robot.occupancy_map_z_min,
z_max=robot.occupancy_map_z_max
Expand Down
2 changes: 1 addition & 1 deletion exts/omni.ext.mobility_gen/omni/ext/mobility_gen/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,4 @@ def to_json(self):

@staticmethod
def from_json(data: str):
return Config(**json.loads(data))
return Config(**json.loads(data))
2 changes: 1 addition & 1 deletion exts/omni.ext.mobility_gen/omni/ext/mobility_gen/extension.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -228,4 +228,4 @@ async def _build_scenario_async():

# self.scenario.save(path)

asyncio.ensure_future(_build_scenario_async())
asyncio.ensure_future(_build_scenario_async())
61 changes: 61 additions & 0 deletions scripts/generate_augmented_scenes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
#!/usr/bin/env python3

# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""Script for generating augmented scenes for an environment."""

import argparse
import os
import subprocess


def main():
parser = argparse.ArgumentParser(description="Generate augmented scenes for an environment")
parser.add_argument("--env_url", type=str, required=True,
help="URL of the environment USD file")
parser.add_argument("--output_dir", type=str, required=True,
help="Directory to save generated scenes")
parser.add_argument("--num_scenes", type=int, default=1,
help="Number of scenes to generate")
parser.add_argument("--num_obstacles", type=int, default=20,
help="Number of obstacles per scene")

args = parser.parse_args()

# Create output directory
os.makedirs(args.output_dir, exist_ok=True)

# Create command for generate_augmented_scenes_implemetation.py
cmd = [
"./app/python.sh",
"scripts/generate_augmented_scenes_implemetation.py",
"--ext-folder", "exts",
"--enable", "omni.ext.mobility_gen",
"--enable", "isaacsim.asset.gen.omap",
f"--env_url={args.env_url}",
f"--output_dir={args.output_dir}",
f"--num_scenes={args.num_scenes}",
f"--num_obstacles={args.num_obstacles}"
]

# Run the command
print(f"\nProcessing environment...")
print(f"Command: {' '.join(cmd)}")
subprocess.run(cmd)


if __name__ == "__main__":
main()
Loading