feat(evaluation): add VLMMetrics#545

Open

davidberenstein1957 wants to merge 38 commits intomainfrom

feat/metrics-vlm-support

Commits on Mar 19, 2026

feat(evaluation): add VLM-based metrics with litellm and transformers support
davidberenstein1957
committed
fix(evaluation): ARNIQA not in torchmetrics - implement manually
davidberenstein1957
committed
fix(evaluation): use List-based scores pattern matching Pruna standards
davidberenstein1957
committed
fix(evaluation): use sync completion instead of async acompletion
davidberenstein1957
committed
chore(evaluation): remove ARNIQA from VLM PR - has dedicated PR #547
davidberenstein1957
committed
feat(evaluation): add structured generation to VLM metrics
davidberenstein1957
committed
fix(evaluation): fix linting issues in VLM metrics
davidberenstein1957
committed
fix(evaluation): fix remaining linting issues
davidberenstein1957
committed
fix(evaluation): fix D205 docstring issues in VLM classes
davidberenstein1957
committed
fix(evaluation): fix import sorting in __init__.py
davidberenstein1957
committed
fix(evaluation): skip docstring check for metrics_vlm
davidberenstein1957
committed
fix(evaluation): enhance docstrings for VLM metrics and base classes
davidberenstein1957
committed
feat(evaluation): introduce new VLM metrics and integration tests
davidberenstein1957
committed
Delete docs/VLM_METRICS_PROMPT_COMPARISON.md
davidberenstein1957
committed
feat(metrics): paper docstring fixes, VQA use_probability default, vlm docstrings
davidberenstein1957
committed
feat(metrics): enhance metric classes with update and compute docstrings
davidberenstein1957
committed
fix(vlm_base): update response_format type hints for clarity
davidberenstein1957
committed
refactor(vlm_base): simplify response_format check for pydantic usage
davidberenstein1957
committed
fix(vlm_base): add "json" option to response_format type hints
davidberenstein1957
committed
feat(dependencies): add pruna[evaluation] to dev dependencies
davidberenstein1957
committed
refactor(metrics): improve docstring consistency and formatting across metric classes
davidberenstein1957
committed
refactor(metrics): update response formats and improve utility functions
davidberenstein1957
committed
refactor(metrics): update collation functions and enhance benchmark task creation
davidberenstein1957
committed
refactor(data): update seed parameter handling and add warnings for test-only benchmarks
davidberenstein1957
committed

Commits on Mar 25, 2026

Limit CPU fallback to VLM metrics
davidberenstein1957
committed

Commits on Mar 26, 2026

Fix QA accuracy litellm test input
davidberenstein1957
committed