The official repository introducing MaskPose and BBox-Mask-Pose methods.
This repository provides the code and model weights for the paper Detection, Segmentation, and Pose Estimation for Multiple Bodies: Closing the Virtuous Circle.
The code is a modified version of MMPose 2.0 with the following key changes:
- Support for multi-dataset training of ViTPose, previously implemented in the official ViTPose repository but absent in MMPose.
- Added
MaskBackgrounddata augmentation to train MaskPose. - Includes weights for MaskPose and RTMDet to reproduce BBox-Mask-Pose results from the paper.
Note: The code has not undergone extensive cross-platform testing and may contain bugs. If you encounter issues, please report them via the repository's issue tracker.
If you use this work, kindly cite it using the reference provided below.
- 02 Dec 2024: The code is available
- 23 Nov 2024: The project website is on
The code builds on MMPose. Install its dependencies simply with:
pip install mmengine
mim install "mmcv>=2.0.1"
mim install "mmdet>=3.1.0"And then install our code using
pip install -r requirements.txt
pip install -e .Results on COCO val and OCHuman val of different Human Pose Estimation (HPE) methods. All results are with detection from RTMDet-l from MMDetection.
We provide trained weights for both MaskPose (introduced in the paper) and ViTPose trained in the multi-dataset setup. Multi-dataset ViTPose is not new but their weights are not compatible with popular MMPose 2.0 codebase. We retrained theme in the MMPose 2.0 environment.
| Model | Datasets | COCO AP | OCHuman AP | weights | notes |
|---|---|---|---|---|---|
| ViTPose-b | COCO+AIC+MPII | 76.3 | 42.5 | download | multi-dataset training compatible with MMPose 2.0 |
| MaskPose-b | COCO+AIC+MPII | 76.4 | 45.3 | download |
To run BBox-Mask-Pose loop, you also need to adapt a detector. We fine-tuned RTMDet-l with mask-out data augmentation as shown in the paper. Weights of the fine-tuned model (compatible with MMDetection config) is here.
Please read carefully the terms and conditions and any accompanying documentation before you download and/or use the RePoGen model, data and software, (the "Model & Software"). By downloading and/or using the Model & Software (including downloading, cloning, installing, and any other use of this github repository), you acknowledge that you have read these terms and conditions, understand them, and agree to be bound by them. If you do not agree with these terms and conditions, you must not download and/or use the Model & Software. Any infringement of the terms of this agreement will automatically terminate your rights under this License.
The code combines MMDetection, MMPose 2.0 and ViTPose.
The code was implemented by Miroslav Purkrábek.
For questions, please use the Issues of Discussion.
BibTeX will be here once submitted to ArXiv