Warning
The repo is still under heavy development and may not work as expected.
The repo allows one to attach a simulator to a Gaussian splat, in this case the Drake Robotics Toolbox to "simulate" the Gaussian splat. This can be helpful in automated data collection, especially of visually realistic camera data that is often necessary to train diffusion models.
The repo relies on the Gaussian splat trained using the nerfstudio library. Follow the instructions here.
The repository can be easily accessed using pixi, an alternative to conda and pip. Installation instructions can be seen here.
Note
If you want to run the example, you can skip step-1.
Once the Gaussian splat is trained and placed in the assets directory, run the match_splat.py as a notebook file to segment the elements in your Gaussian splat that are required to be connected to the simulator. In this repo an example is provided with a pre-trained environment. Upon extracting the necessary masks from the previous step, proceed to the next step.
Now run the demo code to visualise the example. Two browser windows can be opened beside each other, one for viser, and another for the meshcat visualiser from Drake.
pixi r python examples/demo_pusht_splat.py
Upon running, you should see an interface similar to the one shown below,
The demo uses Microsoft Edge browser with a split window functionality to view the two viewers simultaneously.
The "digital twin" environment is wrapped as an env compatible with Gymnasium, meaning, that the Guassian splat can be used to perform training or for collecting data using the typical RL pipeline.
- The repo uses the code from splatnav for loading and viewing the Gaussian splat.
- The communication pipeline between Drake and the Gaussian splat is inspired from the rtxdrake project.
- The pushT task and the teleoperation interface is implemented from the diffusion-policy repository.
- Similar projects: SplatSim and Splat-Sim.