[Paper] Remedy-R: Generative Reasoning for Machine Translation Evaluation without Error Annotations

Source: arXiv - 2512.18906v1

Overview

The paper presents Remedy‑R, a new machine‑translation (MT) evaluation metric that generates a human‑readable reasoning trace before outputting a quality score. Trained only on pairwise preference data (no error‑span annotations or LLM distillation), Remedy‑R matches or exceeds state‑of‑the‑art scalar metrics and even GPT‑4‑based judges on recent WMT benchmarks while offering far more interpretability and robustness to out‑of‑distribution inputs.

Key Contributions

• Generative, reasoning‑driven metric: Produces step‑by‑step analyses of accuracy, fluency, and completeness, followed by a final numeric score.
• Preference‑only training: Learns from 60 K translation‑pair preferences across two language pairs, eliminating the need for costly error‑span annotations.
• Competitive performance: Achieves parity with top scalar metrics and GPT‑4 judges on WMT22‑24 meta‑evaluation, and generalizes well to unseen language pairs.
• Robustness to OOD stress tests: Shows stable behavior on noisy, domain‑shifted, and adversarial translation inputs.
• Self‑reflective feedback loop: The generated analysis can be fed back to a translation model, forming the Remedy‑R Agent that iteratively improves translations.
• Open‑source‑friendly design: No reliance on closed‑source LLMs for distillation, making the approach reproducible for the community.

Methodology

1. Data Collection – The authors gathered 60 K translation pairs with human preference labels (which translation is better) for English↔German and English↔Japanese.
2. Model Architecture – A decoder‑only transformer (similar in size to LLaMA‑7B) is fine‑tuned to take a source sentence, two candidate translations, and output a structured reasoning chain:
   • Accuracy check (does the translation convey the source meaning?)
   • Fluency check (is the target language natural?)
   • Completeness check (are all source content elements present?)
   • Final score (0–100).
3. Reinforcement Learning from Preferences (RLHF‑style) – Using the pairwise preferences, the model is rewarded when its final score ranks the preferred translation higher. The reasoning steps are not directly supervised; they emerge as the model learns to justify its ranking.
4. Self‑Reflection & Revision – For the Remedy‑R Agent, the reasoning output is parsed to identify weak spots (e.g., “missing ‘date’ information”). A downstream translation model is prompted with this feedback to regenerate a better candidate, which is re‑evaluated iteratively.

The pipeline stays lightweight: a single forward pass yields both an interpretable analysis and a numeric metric, avoiding separate error‑detection modules.

Results & Findings

• Interpretability: Human evaluators rated Remedy‑R’s reasoning as “clearly useful” in 84 % of cases, whereas black‑box metrics offered no insight.
• Agent performance: Applying the evaluate‑revise loop improved BLEU scores by 1.2–2.5 points across four translation back‑ends (Qwen2.5, ALMA‑R, GPT‑4o‑mini, Gemini‑Flash).
• Efficiency: Inference time per sentence ≈ 120 ms on a single A100 GPU, comparable to COMET‑22.

Practical Implications

• Debugging translations: Developers can surface concrete error categories (missing entities, awkward phrasing) directly from the metric, speeding up QA cycles.
• Automated post‑editing: The Remedy‑R Agent can be integrated into CI pipelines to automatically polish model outputs before deployment, reducing manual post‑editing costs.
• Model‑agnostic evaluation: Because the metric does not depend on a specific translation system, it can serve as a universal “oracle” for benchmarking new MT models or for continuous monitoring in production.
• Low‑resource adaptability: Training only on preference data means teams can bootstrap a reasoning metric for niche language pairs with modest annotation effort.
• Safety & robustness: The reasoning trace helps flag OOD failures (e.g., hallucinations) that scalar scores might miss, supporting more reliable MT services in high‑stakes domains like medical or legal translation.

Limitations & Future Work

• Scale of reasoning: The current model’s reasoning depth is limited to three pre‑defined dimensions (accuracy, fluency, completeness). More nuanced linguistic phenomena (style, register) are not captured.
• Preference data bias: The metric inherits any systematic bias present in the human preference annotations (e.g., over‑valuing fluency over adequacy).
• Language coverage: Experiments focus on two language pairs; while zero‑shot results are promising, broader multilingual validation is needed.
• Agent convergence: The evaluate‑revise loop sometimes plateaus or even degrades quality if the feedback is ambiguous; smarter parsing of the reasoning output could mitigate this.
• Future directions suggested by the authors include: extending the reasoning schema, incorporating multilingual preference datasets, and exploring tighter integration with LLM‑based translators for end‑to‑end trainable pipelines.

Authors

• Shaomu Tan
• Ryosuke Mitani
• Ritvik Choudhary
• Qiyu Wu
• Toshiyuki Sekiya
• Christof Monz

Paper Information

• arXiv ID: 2512.18906v1
• Categories: cs.CL
• Published: December 21, 2025
• PDF: Download PDF