Abstract
+Despite the steady progress in machine translation evaluation, existing automatic metrics struggle to capture how well meaning is preserved beyond sentence boundaries. We posit that reliance on a single intrinsic quality score, trained to mimic human judgments, might be insufficient for evaluating translations of long, complex passages, and a more “pragmatic” approach that assesses how accurately key information is conveyed by a translation in context is needed. We introduce TREQA (Translation Evaluation via Question-Answering), a framework that extrinsically evaluates translation quality by assessing how accurately candidate translations answer reading comprehension questions that target key information in the original source or reference texts...
+The TREQA Framework
+ }})
+ Figure 1: TREQA assesses translation quality through a question-answering framework.
+Try TREQA Live
+This demo uses an example from the paper (Figure 6) to show how TREQA identifies subtle translation errors. You can edit the text or add your own candidates.
+