FAIRSET was constructed with the goal of having a publicly available image dataset for the purpose of evaluating the precision and biaises of points placement of Face Landmark Detection (FLD) artificial intelligence models. Few models provide information about their biases and even less research is conducted about it, specifically for face landmarking models.
FAIRSET images are sourced from WiderFace and DAD-3DHeads datasets, and the base of the demographics annotations are taken from Amazon Alexa WiderFace and DAD-3DHeads annotations.
FAIRSET annotations contain information about:
- face landmarks;
- skin tone (Fitzpatrick scale);
- perceived sex;
- age (child, young adult, adult and senior).
The annotations are available in fairset.json. Additional bounding box information for targetted faces is available in fairset_bbox.json. The original bounding boxes from WiderFace and DAD-3DHeads were corrected using the RetinaFace model.
An example using FAIRSET for the evaluation of biases of Google MediaPipe FaceMesh is provided here
For additional information on FLD biaises, take a look Benchmarking Facial Landmarks Estimation: Evaluating Popular Algorithms Using FAIRSET, a Balanced Landmark Database, Joly et al., in the IGS 2025 conference proceedings, where we present results about MediaPipe FaceMesh, OpenFace 2.2 and 3DDFAv3.
The provided code was tested on Python 3.11, but it might still work on versions >= 3.9
- Python 3.11
python3 -m pip install -r ./requirements.txt
- Request access to the DAD-3DHeads dataset: https://www.pinatafarm.com/research/dad-3dheads/dataset
- Execute the download script
Example:
python3 ./download.py -o assets -d ~/DAD-3DHeadsDataset.zip
usage: python3 download.py -d DAD3dHEADS [-h] [-a] [-r] [-f] [-w WIDERFACE [WIDERFACE ...]] [-o OUTPUT]
options:
- -h, --help Show this help message and exit
- -d DAD3dHEADS, --dad3d DAD3dHEADS Specify the location of the zip, or the download link, of the DAD-3DHeads dataset
⚠️ passing the direct download link instead of specifying the local zip might not work (missing header from AWS)- -a, --alexa Download the Amazon Alexa demographics annotations for the WiderFace subset
- -r, --regenerate Regenerate the merged Amazon Alexa / Widerface annotations, if downloaded. Can be used with -a/--alexa
- -w WIDERFACE [WIDERFACE ...], --widerface WIDERFACE [WIDERFACE ...]
(Optional) Specify the location of the widerface zip(s). If no zip is specified, this script will try to fetch and extract the specific images from the remote zips on HuggingFace.
⚠️ HuggingFace might throttle you if you execute this script multiple times. In that case, download the zip(s) locally and pass zip the location(s) using -w.- -f, --force Force download the dataset even if it seems present
- -o OUTPUT, --output OUTPUT Directory to download the dataset to.
Default: assets
This download script will create the following subdirectories:
- FAIRSET
- alexa (if -a was passed)
Analysis steps and details can be found in the analysis README.
A Python exemple is provided analysing the biases of MediaPipe FaceMesh.
Dad-3DHeads
T. Martyniuk, O. Kupyn, Y. Kurlyak, I. Krashenyi, J. Matas, and V. Sharmanska, “DAD-3DHeads: A Large-scale Dense, Accurate and Diverse Dataset for 3D Head Alignment from a Single Image,” in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA: IEEE, Jun. 2022, pp. 20910–20920. doi: 10.1109/CVPR52688.2022.02027.Amazon Alexa Widerface Annotations
Y. Yang et al., “Enhancing Fairness in Face Detection in Computer Vision Systems by Demographic Bias Mitigation,” in Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, Oxford United Kingdom: ACM, Jul. 2022, pp. 813–822. doi: 10.1145/3514094.3534153, https://www.amazon.science/publications/enhancing-fairness-in-face-detection-in-computer-vision-systems-by-demographic-bias-mitigation
Please cite the following papers if you use FAIRSET in your research projects :
FAIRSET dataset
@inproceedings{fairset2025,
title = {Towards a Gold Standard for AI Facial Landmarks Estimation: Constructing FAIRSET, a Balanced and Inclusive Landmark Database},
author = {Zelovic, Nikola and Joly, Ian-Mathieu and Riesco Eléonor and Lebel, Karina},
year = 2026,
month = {January},
booktitle = {Proceedings IGS 2025 - Investigating Human Movements | Handwriting and Beyond},
publisher = {Les Presses de l'Université de Montréal (PUM)},
address = {Montreal, QC, Canada},
pages = {218--221},
note = {Available at \url{https://pum.umontreal.ca/catalogue/proceedings_igs_2025}},
editor = {Réjean Plamondon, Céline Rémi, Karina Lebel, Mickaël Begon, Frederic Fol Leymarie, Lama Séoud, Benjamin de Leener, Eva Alonso Ortiz},
organization = {International Graphonomics Society}
}
Face Landmark Detection AI Evaluation and Bias Analysis Code
@inproceedings{joly_benchmarking_2025,
title = {Benchmarking Facial Landmarks Estimation: Evaluating Popular Algorithms Using FAIRSET, a Balanced Landmark Database},
author = {Joly, Ian-Mathieu and Zelovic, Nikola and Riesco Eléonor and Lebel, Karina},
year = 2026,
month = {January},
booktitle = {Proceedings IGS 2025 - Investigating Human Movements | Handwriting and Beyond},
publisher = {Les Presses de l'Université de Montréal (PUM)},
address = {Montreal, QC, Canada},
pages = {93--96},
note = {Available at \url{https://pum.umontreal.ca/catalogue/proceedings_igs_2025}},
editor = {Réjean Plamondon, Céline Rémi, Karina Lebel, Mickaël Begon, Frederic Fol Leymarie, Lama Séoud, Benjamin de Leener, Eva Alonso Ortiz},
organization = {International Graphonomics Society}
}
FAIRSET is licensed under CC BY-NC-SA 4.0. View the LICENSE file or visit https://creativecommons.org/licenses/by-nc-sa/4.0/