From ScanDDM to ART: DDM-DINO

Visual attention models have demonstrated a growing capability in predicting scanpaths, which are sequences of fixations and eye movements. Specifically, ScanDDM introduced a DDM-based approach for predicting goal-directed scanpaths in a zero-shot modality, while ART focused on the incremental prediction of attention during language-guided object referral tasks. The present work explores the combination of these two approaches, modifying ScanDDM with the integration of GroundingDINO to address the incremental object referral task. The resulting model has been named DDM-DINO.

More detailed informations and examples of usage can be found in the attached PDF report.

Setup

Install all the requirements with pip install -r requirements.txt

Usage

1. In main.py define the prompt and the image path
2. Run python main.py

Metrics

1. Uncomment the commented libraries in requirements.txt
2. Install the new requirements with pip install -r requirements.txt
3. In calculate_all_metrics.py define the parameters
4. Run python calculate_all_metrics.py

---

Project for Natural Interaction and Affective Computing courses, UNIMI @ PHuSe Lab, AY 2024/2025, by Hari Calzi and Salvatore Ferrara.