[AAAI 2025] QORT-Former: Query-optimized Real-time Transformer for Understanding Two Hands Manipulating Objects

Elkhan Ismayilzada*, MD Khalequzzaman Chowdhury Sayem*, Yihalem Yimolal Tiruneh, Mubarrat Tajoar Chowdhury, Muhammadjon Boboev, Seungryul Baek

_{* Equal contribution.}

Installation

git clone https://github.com/kcsayem/QORT-Former.git

Create a conda environment

conda create -n "qort_former" python=3.10

Install PyTorch

conda install pytorch==1.13.0 torchvision==0.14.0 torchaudio==0.13.0 pytorch-cuda=11.6 -c pytorch -c nvidia

Install requirements

pip install -r requirements.txt

Data Preparation

Download FPHA, H2O datasets.

Pretrained weights

Pretrained weights are available for H2O and FPHA ($${\color{green}\text{Coming soon...}}$$).

Inference

python main.py --model_path {path_to_model} --source {image_folder}

The predictions will be generated and saved in the {image_folder}/predictions directory.

Acknowledgements

This repository is based on

• FastInst
• H2OTR
• GraFormer
• DETR

Citing

@inproceedings{ismayilzada2025qort,
  title={QORT-Former: Query-optimized Real-time Transformer for Understanding Two Hands Manipulating Objects},
  author={Ismayilzada, Elkhan and Sayem, MD Khalequzzaman Chowdhury and Tiruneh, Yihalem Yimolal and Chowdhury, Mubarrat Tajoar and Boboev, Muhammadjon and Baek, Seungryul},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={39},
  number={4},
  pages={3895--3903},
  year={2025}
}