Add BARTScore metric (NeurIPS 2021)

[vkehfdl1]

Description

Implement BARTScore (Yuan et al., NeurIPS 2021) as a generation evaluation metric.

What it measures

Directional semantic evaluation: faithfulness, precision, recall, F1.

How it works

• Uses pretrained BART model to compute log-probability of one text given another
• Three configurations:
  • BARTScore(context → answer): faithfulness to retrieved context
  • BARTScore(reference → answer): precision
  • BARTScore(answer → reference): recall
• Information-theoretic grounding (conditional log-likelihood)
• Deterministic, no LLM-as-judge

Why

• Published at NeurIPS — strong venue credibility
• Provides directional evaluation (faithfulness vs precision vs recall)
• Complements AlignScore (NLI-based) with probability-based approach → triangulation
• NeurIPS 2026 ED Track submission

Reference

Yuan et al., "BARTScore: Evaluating Generated Text as Text Generation," NeurIPS 2021

[vkehfdl1]

• AI-understood issue summary
  BARTScore would add a deterministic, model-based generation metric that scores one text conditioned on another via BART log-likelihood. In this repo, it maps cleanly to three directional evaluation views already implied by MetricInput: context -> answer for faithfulness (using retrieved_contents), reference -> answer for precision, and answer -> reference for recall, with an optional F1 summary.

• Why this issue is needed

  • The current generation metric set already covers lexical overlap and embedding similarity, but it does not yet expose a first-class probability-based directional scorer.
  • That makes this a good complement to existing semantic metrics: it gives a different signal than ROUGE/BERTScore/SemScore and strengthens metric triangulation for the NeurIPS 2026 ED-track motivation described here.
  • It is also deterministic and avoids LLM-as-judge cost/variance.

• Why this issue may not be needed

  • There is some overlap with existing semantic metrics, so the incremental value depends on whether maintainers specifically want a directional conditional-likelihood signal rather than another similarity score.
  • The implementation is heavier than the current lightweight metrics because it likely adds a larger pretrained seq2seq inference path and associated test/runtime burden.
  • The faithfulness variant needs one explicit repo-level decision up front: how retrieved_contents: list[str] should be concatenated/truncated before scoring. Without a stable policy, cross-run and cross-pipeline comparisons may be noisy.
  • If maintainers want to keep core dependencies lean, this could reasonably start as a plugin before being promoted into core.

• Expected Outcome

  • A BARTScoreConfig + generation metric implementation following the existing generation.py metric/config pattern.
  • Supported modes for faithfulness, precision, recall, and optional f1.
  • A documented, deterministic mapping from repo inputs to BARTScore source/target texts.
  • Config YAML, exports, docs, and tests consistent with the existing generation-metric surface.

• Concise implementation plan

  1. Mirror the existing built-in generation metric pattern in autorag_research/evaluation/metrics/generation.py, evaluation/metrics/__init__.py, and configs/metrics/generation/.
  2. Choose the implementation surface carefully: either wrap the canonical BARTScore approach directly or provide a minimal local scorer built on the repo's existing HF/transformers dependency surface.
  3. Define the text mapping explicitly:
     • context -> answer: joined/truncated retrieved_contents vs generated_texts
     • reference -> answer: generation_gt vs generated_texts
     • answer -> reference: generated_texts vs generation_gt
     • f1: harmonic mean of precision/recall
  4. Add tests by following the current bert_score / semantic-metric patterns in tests/autorag_research/evaluation/metrics/test_generation.py.
  5. Add docs explaining directionality, model/runtime requirements, and the context-joining policy so users know what signal each mode represents.

• Agent Signature

[vkehfdl1]

/approve