🏐 VolleyBots: A Testbed for Multi-Drone Volleyball Game Combining Motion Control and Strategic Play

Zelai Xu*, Ruize Zhang*, Chao Yu+, Huining Yuan, Xiangmin Yi, Shilong Ji, Chuqi Wang, Wenhao Tang, Feng Gao, Wenbo Ding, Xinlei Chen, Yu Wang+

   

News | Overview | Installation | Usage | Checkpoints | Experimental Setup | Citation | Acknowledgement

📣 News

[2025.11.19] 🔥 We open-source the HCSP code.

[2025.09.29] 🔥 JuggleRL details how we sim‑to‑real on the Solo Bump task and has been released on arXiv.

[2025.09.19] 🎉 Our testbed is accepted to NeurIPS 2025 Datasets & Benchmarks Track.

[2025.08.09] 🎉 HCSP proposes a hierarchical RL framework to tackle the 3 vs 3 task and has been accepted to CoRL 2025.

[2025.06.03] Add a full 6 vs 6 volleyball task, new RL & MARL baselines, a hierarchical policy for the 3 vs 3 task, and extra reproducible training/evaluation shells and checkpoints. (v0.1.0)

[2025.02.09] Initial release with 8 tasks, 8 baselines, and reproducible training/evaluation shells & checkpoints.

📌 Overview

Overview of the VolleyBots Testbed. VolleyBots comprises three key components: (1) Environment, supported by Isaac Sim and PyTorch, which defines entities, observations, actions, and reward functions; (2) Tasks, including 3 single-agent tasks, 3 multi-agent cooperative tasks and 3 multi-agent competitive tasks; and (3) Algorithms, encompassing RL, MARL, game-theoretic algorithms.

⚙️ Installation

We use Isaac Sim as the simulation environment. You can choose to install the local version or the container version.

Option 1: Install Local Version

1. Isaac Sim

Download the Omniverse Isaac Sim and install the desired Isaac Sim release following the official document.

For this repository, we use the Isaac Sim version 2023.1.0-hotfix.1. Since this version is not available on the official website, you can download it directly from here. After downloading, extract the file and move the folder to the following directory:

mv isaac_sim-2023.1.0-hotfix.1 ~/.local/share/ov/pkg/

Set the following environment variables to your ~/.bashrc or ~/.zshrc files :

# Isaac Sim root directory
export ISAACSIM_PATH="${HOME}/.local/share/ov/pkg/isaac_sim-2023.1.0-hotfix.1"

After adding the environment variable, apply the changes by running:

source ~/.bashrc

2. Conda

Although Isaac Sim comes with a built-in Python environment, we recommend using a seperate conda environment which is more flexible. We provide scripts to automate environment setup when activating/deactivating a conda environment at VolleyBots/conda_setup.

conda create -n volley python=3.10
conda activate volley

# at VolleyBots/
cp -r conda_setup/etc $CONDA_PREFIX
# re-activate the environment
conda activate volley
# install VolleyBots
pip install -e .

# verification
python -c "from omni.isaac.kit import SimulationApp"
# which torch is being used
python -c "import torch; print(torch.__path__)"

3. Third Party Packages

VolleyBots requires specific versions of the tensordict, torchrl and orbit packages. For this repository, we manage these three packages using Git submodules to ensure that the correct versions are used. To initialize and update the submodules, follow these steps:

Get the submodules:

# at VolleyBots/
git submodule update --init --recursive

Install tensordict:

# at VolleyBots/
cd third-party/tensordict
python setup.py develop

Before install torchrl, first check and update gcc and g++:

# check gcc version, should be like: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0 ...
gcc --version

# if not gcc 9.x, check available gcc
ls /usr/bin/gcc*

# if gcc-9 is not installed
sudo apt update && sudo apt install gcc-9
# if gcc-9 is installed
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-9 100
sudo update-alternatives --config gcc
# then follow instructions to select gcc-9.

# check gcc version again
gcc --version

# apply same update and config to g++

Then install torchrl:

# at VolleyBots/
cd third-party/rl
python setup.py develop

Also we need to install orbit package of Isaac Sim. Note that currently orbit package has been integrated into Isaac Lab package, but this branch still uses the standalone orbit package (this can be updated in the future). So we manage the older version of orbit package using Git submodule. To install the orbit package, follow these steps:

# at VolleyBots/
cd third-party/orbit
# create a symbolic link
ln -s ${ISAACSIM_PATH} _isaac_sim

# create environment variable
echo -e "alias orbit=$(pwd)/orbit.sh" >> ${HOME}/.bashrc
source ${HOME}/.bashrc

# building extentions
sudo apt install cmake build-essential
./orbit.sh --install  # or "./orbit.sh -i"

4. Verification

# at VolleyBots/
cd scripts
python train.py headless=true wandb.mode=disabled total_frames=50000 task=SingleJuggleVolleyball task.env.num_envs=16

Option 2: Install Container Version (using Docker)

The Container version is easier to set up compared to the Local version. However, it's important to note that the Container version does not support real-time rendering. Therefore, it only supports the command with headless=true. You can save videos during training and upload them to Wandb.

First, make sure your computer has installed Docker, NVIDIA Driver and NVIDIA Container Toolkit. Then, you should successfully run:

# Verification
docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

Download the image from Docker Hub:

docker pull jimmyzhangruize/isaac-sim:2023.1.0-hotfix.1

This image already includes Isaac sim, so you don't neet to download Isaac sim. However, you need to clone the VolleyBots repository to your local computer because we use VolleyBots mounted in the container.

Then, run the image:

docker run --name *** --entrypoint bash -dit --gpus all -e "ACCEPT_EULA=Y" --rm --network=host \
-e "PRIVACY_CONSENT=Y" \
-v ~/docker/isaac-sim/cache/kit:/isaac-sim/kit/cache:rw \
-v ~/docker/isaac-sim/cache/ov:/root/.cache/ov:rw \
-v ~/docker/isaac-sim/cache/pip:/root/.cache/pip:rw \
-v ~/docker/isaac-sim/cache/glcache:/root/.cache/nvidia/GLCache:rw \
-v ~/docker/isaac-sim/cache/computecache:/root/.nv/ComputeCache:rw \
-v ~/docker/isaac-sim/logs:/root/.nvidia-omniverse/logs:rw \
-v ~/docker/isaac-sim/data:/root/.local/share/ov/data:rw \
-v ~/docker/isaac-sim/documents:/root/Documents:rw \
-v ***:/root/VolleyBots:rw \
-v /data:/data \
-e "WANDB_API_KEY=***" \
jimmyzhangruize/isaac-sim:2023.1.0-hotfix.1

Note that:

1. You need to replace *** in --name *** with the your docker image name.
2. You need to replace *** in -v ***:/root/VolleyBots:rw with the directory where VolleyBots is locally located in.
3. You need to replace *** in -e "WANDB_API_KEY=***" with your own WANDB_API_KEY. If you do not need to use Wandb, you can omit this line.
4. In the container, VolleyBots is located at /root/VolleyBots and Isaac-sim is located at /isaac-sim.

Install VolleyBots in the container:

conda activate sim2
cd /root/VolleyBots
cp -r conda_setup/etc $CONDA_PREFIX
conda activate sim2 # re-activate the environment
pip install -e . # install VolleyBots

Verify you can successfully run VolleyBots in the container (use deploy branch):

cd /root/VolleyBots/scripts
python train.py headless=true wandb.mode=disabled total_frames=50000 task=SingleJuggleVolleyball task.env.num_envs=16

🚀 Usage

1. Configuration

We use hydra to manage the configurations. You can find the configuration files in cfg.

File structure of the configuration files:

cfg
├── train.yaml # training configurations
├── task
│   ├── <TASKNAME>.yaml # configurations for each task
├── algo 
│   ├── <ALGONAME>.yaml # configurations for each algorithm

When you run the training script, you can modify the configurations in these yaml files, or you can pass the configurations as command line arguments. For example:

python train.py headless=true \
    total_frames=1_000_000_000 \
    task=Volleyball1v1 \
    task.drone_model=Iris \
    task.env.num_envs=2048 \
    task.env.env_spacing=18 \
    task.random_turn=True \
    task.initial.drone_0_near_side=True \
    task.symmetric_obs=True \
    eval_interval=300 \
    save_interval=300 \
    algo=mappo \
    algo.share_actor=true \
    algo.critic_input="obs" \
    seed=0 \

For experiment tracking, we use wandb. You need to have a wandb account, and set the wandb configurations in the train.yaml file. For example:

wandb:
  group: <EXPERIMENT_GROUP>
  run_name: <EXPERIMENT_NAME>
  job_type: train
  entity: <YOUR_WANDB_ENTITY>
  project: <YOUR_WANDB_PROJECT>
  mode: online # set to 'disabled' when debugging locally
  run_id:
  monitor_gym: True
  tags:

2. Training

To train a model, you can run the following command:

# at VolleyBots/scripts
python train.py <TRAIN_CONFIGS> <TASK_CONFIGS> <ALGO_CONFIGS>

We also provide example training scripts for each task included in our paper. You can find them in the scripts/training_shell directory. For example, to train a model for the Volleyball1v1 task using self-play, you can run:

# at VolleyBots/scripts/training_shell
bash volleyball1v1_sp.sh

3. Evaluation

We include the example evaluation scripts from our paper in the scripts/evalution_shell directory. For example, to evalute the cross-play win rate between psro_uniform and psro_nash on the Volleyball3v3 task, you can run:

# at VolleyBots/scripts/evaluation_shell
bash volleyball3v3_crossplay.sh

🤖 Checkpoints

We offers trained checkpoints at VolleyBots/checkpoints directory for evaluating exploitability and cross-play win rates, and we also include hierarchical policy checkpoints for competing against the self-play baseline in the 3 vs 3 task.

💻 Experimental Setup

Our experiments were run on a workstation with:

• GPU: NVIDIA GeForce RTX 4090
• RAM: 128 GB
• OS: Ubuntu 20.04 LTS
• Software: CUDA 12.4

📚 Citation

Please cite our paper if you find this testbed useful:

@misc{xu2025volleybotstestbedmultidronevolleyball,
      title={VolleyBots: A Testbed for Multi-Drone Volleyball Game Combining Motion Control and Strategic Play}, 
      author={Zelai Xu and Ruize Zhang and Chao Yu and Huining Yuan and Xiangmin Yi and Shilong Ji and Chuqi Wang and Wenhao Tang and Feng Gao and Wenbo Ding and Xinlei Chen and Yu Wang},
      year={2025},
      eprint={2502.01932},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2502.01932}, 
}

Moreover, if you reference our 3 vs 3 hierarchical learning algorithm (HCSP) or our sim-to-real transfer work for the Solo Bump task (JuggleRL), please also cite:

@misc{zhang2025masteringmultidronevolleyballhierarchical,
      title={Mastering Multi-Drone Volleyball through Hierarchical Co-Self-Play Reinforcement Learning}, 
      author={Ruize Zhang and Sirui Xiang and Zelai Xu and Feng Gao and Shilong Ji and Wenhao Tang and Wenbo Ding and Chao Yu and Yu Wang},
      year={2025},
      eprint={2505.04317},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2505.04317}, 
}

@misc{ji2025jugglerlmasteringballjuggling,
      title={JuggleRL: Mastering Ball Juggling with a Quadrotor via Deep Reinforcement Learning}, 
      author={Shilong Ji and Yinuo Chen and Chuqi Wang and Jiayu Chen and Ruize Zhang and Feng Gao and Wenhao Tang and Shu'ang Yu and Sirui Xiang and Xinlei Chen and Chao Yu and Yu Wang},
      year={2025},
      eprint={2509.24892},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2509.24892}, 
}

🙏 Acknowledgement

This repository is heavily based on OmniDrones.

Some of the abstractions and implementation was heavily inspired by Isaac Orbit.