Add AlignScore metric (ACL 2023)

[vkehfdl1]

Description

Implement AlignScore (Zha et al., ACL 2023) as a generation evaluation metric.

What it measures

Factual consistency / faithfulness of generated text to source context.

How it works

• RoBERTa-based model trained on 15 datasets (4.7M examples): NLI, fact verification, QA, paraphrase
• Splits generated text into chunks, scores each against source, aggregates
• Outperforms GPT-3.5-based evaluation on SummaC and TRUE benchmarks
• Fully deterministic, no LLM-as-judge

Why

• Replaces RAGAS Faithfulness with academically rigorous alternative
• Reference-free (evaluates against retrieved context only)
• NeurIPS 2026 ED Track submission

Reference

Zha et al., "AlignScore: Evaluating Factual Consistency with a Unified Alignment Function," ACL 2023

[vkehfdl1]

• AI-understood issue summary
  This issue asks for an AlignScore-based generation metric that scores whether a model answer is factually consistent with the retrieved/source context, giving AutoRAG-Research a deterministic, reference-free faithfulness metric instead of an LLM-judge-style proxy. In the current repo, the closest existing options are the built-in response_relevancy metric and the separate Trust-Align plugin metrics; there is no AlignScore implementation yet.

• Why this issue is needed
  Faithfulness is a core RAG quality signal, and AlignScore is a stronger fit than query-relevance-only scoring when the goal is to detect hallucinated or unsupported claims. It also aligns well with this repository's metric architecture because the needed inputs already exist conceptually (generated_texts + retrieved/source context), and the project already supports optional model-backed evaluation paths.

• Why this issue may not be needed
  If the immediate need is simply "a deterministic non-LLM factuality metric," this may be better shipped as a plugin first rather than a core built-in metric. The repo already has a plugin path for heavier generation metrics (for example the Trust-Align metrics), and AlignScore will likely add model-weight/runtime maintenance, device handling, and chunk/aggregation semantics that are heavier than the current built-ins.

• Expected Outcome
  A new metric that can be selected in generation evaluation configs, consumes generated answer text plus retrieved/source context, returns a reproducible higher-is-better factual consistency score, and is covered by tests/docs/YAML config the same way other metrics are.

• Concise implementation plan

  1. Confirm the implementation surface: core built-in vs plugin-first, plus official checkpoint/license/runtime assumptions.
  2. Add the metric function + config wrapper in the existing generation-metric pattern, likely using retrieved_contents/source context and generated_texts as inputs.
  3. Expose a YAML config and export surface so the metric is discoverable by experiment configs.
  4. Add deterministic tests by mocking/patching the model scorer, including edge cases such as empty context, long generations, and chunk aggregation behavior.
  5. Add short docs that explain when AlignScore should be preferred over response_relevancy and when plugin deployment is the better fit.

• Agent Signature

[vkehfdl1]

Why it should be plugin? And is your plan to use exact model of the paper? Because the model is so old. (Suggesting replace it to sLLM like gemma 4 and so on)
Plus, what is difference between UniEval metric (EMNLP 2022), MiniCheck metric (EMNLP 2024), and this