Visuotactile-Based Learning for Insertion with Compliant Hands

@article{azulay2025visuotactile,
  title={Visuotactile-Based Learning for Insertion With Compliant Hands},
  author={Azulay, Osher and Ramesh, Dhruv Metha and Curtis, Nimrod and Sintov, Avishai},
  journal={IEEE Robotics and Automation Letters},
  volume={10},
  number={4},
  pages={4053--4060},
  year={2025},
  publisher={IEEE}
}

---

• Overview
• Getting Started
  • Dependencies
  • Installation
• Usage
  • Step 1: Teacher Policy training
  • Step 2: Evaluate the teacher policy
  • Step 3: Visuotactile Student distillation
  • Step 4: Step 4: Evaluate the student
  • Deploy
• License

---

Tactile Rendering in IsaacGym

Built on top of TACTO
This project extends allsight_sim by integrating support for round-shaped tactile rendering for IsaacGym. > For more information about the base simulation framework of the tactile images, please refer to the original allsight_sim project.

---

Overview

This repository provides the official implementation of Visuotactile-Based Learning for Insertion with Compliant Hands.

---

Getting Started

Dependencies

Project was tested on:

• ubuntu >=18.04
• python >= 3.8
• cuda >=11.7
• built on ROS melodic\noetic

Installation

1. First, Create Mamba-Forge environment:

   mamba create -n insertion python=3.8
   mamba activate insertion

2. Install IsaacGym:

   Download the Isaac Gym Preview 4 release from the website, then follow the installation instructions in the documentation.

   Once Isaac Gym is installed and samples work within your current python environment, install this repo:

   pip install -e .

3. Clone and install this repo:

   cd isaacgym/python
   git clone https://github.com/osheraz/IsaacGymInsertion
   cd IsaacGymInsertion && pip install -e .

---

Usage

Brief

• Train a teacher policy using privliged information with RL
• Train a student policy using visual and\or tactile information
• Deploy on real-robot

Note: All configs, logs, and model weights are saved in a folder within outputs/.

As noted before, this repository includes implementations for tactile sensing (built on top of TACTO) and point cloud observation.

To visualize each input, refer to the cfg/task/FactoryTaskInsertionTactile.yaml file and set True for each input you would like to enable.

To visualize tactile inputs, set the following parameters:

tactile=True  
tactile_display_viz=True

You can adjust the tactile parameters in the file allsight\conf\sensor\config_allsight_white.yml.

To visualize depth, point cloud, or masks, set the following parameters:

external_cam=True  
pcl_cam\depth_cam\seg_cam=True 
display=True

Training

Step 1: Teacher Policy training

cd isaacgyminsertion
scripts/train_s1.sh

Step 2: Evaluate the teacher policy

scripts/eval_s1.sh

Step 3: Visuotactile Student distillation

Modify scripts/train_s2.sh given which student policy you want to use:

# for segmented-pcl:
train.ppo.obs_info=True \ train.ppo.img_info=False \ train.ppo.seg_info=False \ train.ppo.pcl_info=True \ train.ppo.tactile_info=False \ task.env.tactile=False \ task.external_cam.external_cam=True \ task.external_cam.depth_cam=False \ task.external_cam.seg_cam=True \ task.external_cam.pcl_cam=True \```

# for segmented-depth:
train.ppo.obs_info=True \ train.ppo.img_info=True \ train.ppo.seg_info=True \ train.ppo.pcl_info=False \ train.ppo.tactile_info=False \ task.env.tactile=False \ task.external_cam.external_cam=True \ task.external_cam.depth_cam=True \ task.external_cam.seg_cam=True \ task.external_cam.pcl_cam=False \```

# for visualtactile (pcl+tactile):
train.ppo.obs_info=True \ train.ppo.img_info=False \ train.ppo.seg_info=False \ train.ppo.pcl_info=True \ train.ppo.tactile_info=True \ task.env.tactile=True \ task.external_cam.external_cam=True \ task.external_cam.depth_cam=False \ task.external_cam.seg_cam=True \ task.external_cam.pcl_cam=True \```

Next, train the policy:

scripts/train_s2.sh

Step 4: Evaluate the student

Evaluate with same arguments in eval_s2.sh:

scripts/eval_s2.sh

---

Deploy

See - deploy_instructions

---

Acknowledgements

• isaacgym
• IsaacGymEnvs-Factory
• dexenv
• hora

---

License

This repository is licensed under the Apache License. Feel free to use, modify, and distribute the code as per the terms of this license.

---

↑ Return