Adding Path MMU Atlas to Eva

[jonaserb-k]

Summary

This PR adds native support for the PathMMU Atlas dataset, a multimodal Visual Question Answering (VQA) benchmark for pathology. The implementation enables eva to directly load and evaluate models on expert-validated pathology questions derived from the ARCH Book Set.

Key changes:

• New PathMMUAtlas dataset class for multimodal VQA evaluation
• Configuration file for running PathMMU Atlas evaluations
• Test suite with test assets
• Added docs for PathMMU Atlas

Dataset Overview

PathMMU Atlas combines:

• Images: 4,270 pathology images from the ARCH Book Set (educational textbooks)
• Questions: Expert-validated VQA questions from the PathMMU HuggingFace dataset

Split	Samples
val	80
test	799
test_tiny	208

Licenses: CC-BY-ND-4.0 (PathMMU), CC-BY-NC-SA-4.0 (ARCH Book Set)

Files Added/Modified

New Files

• src/eva/multimodal/data/datasets/multiple_choice/path_mmu_atlas.py - Main dataset implementation
• configs/multimodal/pathology/online/multiple_choice/path_mmu_atlas.yaml - Evaluation config
• tests/eva/multimodal/data/datasets/multiple_choice/test_path_mmu_atlas.py - Test suite (21 tests)
• tests/eva/assets/multimodal/datasets/path_mmu_atlas/ - Test assets (sample images + data.json)

Modified Files

• src/eva/multimodal/data/datasets/__init__.py - Export PathMMUAtlas
• src/eva/multimodal/data/datasets/multiple_choice/__init__.py - Export PathMMUAtlas
• src/eva/core/metrics/defaults/classification/multiclass.py - Minor fix
• src/eva/vision/data/datasets/classification/__init__.py - Remove duplicate Camelyon16 entry

Testing

• The dataset has been tested with anthropic/claude-3-7-sonnet-20250219 and works fine. More testing is needed to confirm correct workings of the implementation.

Usage

# Run evaluation on PathMMU Atlas
eva test --config configs/multimodal/pathology/online/multiple_choice/path_mmu_atlas.yaml

# With automatic download
DOWNLOAD_DATA=true eva test --config configs/multimodal/pathology/online/multiple_choice/path_mmu_atlas.yaml

[nkaenzig]

Looks great! Left a couple of comments.

[jonaserb-k]

Implemented all changes as requested per PR. I actually removed the whole download.py utilities file since I realised it is too extensive for the actual use case and just implemented everything directly in path_mmu_atlas.py. Also added docs etc.

[nkaenzig]

Solid work 🚀