[Paper] Rubric-Grounded RL: Structured Judge Rewards for Generalizable Reasoning

Source: arXiv - 2605.08061v1

Overview

The paper introduces Rubric‑Grounded Reinforcement Learning (RL), a new way to train language models by breaking down the reward signal into multiple, verifiable criteria (a “rubric”) and letting a frozen large language model (LLM) act as an impartial judge. By rewarding partial credit on each criterion instead of a single binary or holistic score, the authors show that models can learn more robust, generalizable reasoning abilities.

Key Contributions

• Rubric‑grounded reward framework: Formalizes a multi‑criterion reward that is generated by an immutable LLM judge conditioned on external documents the policy never sees.
• Automatic rubric extraction: Derives task‑specific rubrics from a 100 k‑document corpus of scientific and technical texts (OSTI).
• GRPO training pipeline: Applies Group Relative Policy Optimization (GRPO) to fine‑tune Llama‑3.1‑8B‑Instruct using the rubric‑grounded rewards.
• Empirical gains: Achieves 71.7 % normalized reward on a held‑out rubric evaluation and improves performance on four unrelated reasoning benchmarks (GSM8K, MATH, GPQA‑Main, GPQA‑Diamond).
• Evidence of transferability: Demonstrates that structured, document‑grounded rewards can induce reasoning skills that generalize beyond the training corpus.

Methodology

1. Rubric creation

   • The authors parse ~100 k scientific/technical documents to extract criteria (e.g., correctness, completeness, citation quality).
   • Each criterion is assigned a weight reflecting its importance for the target task.

2. LLM judge

   • A large, frozen LLM (the “judge”) receives a model’s response plus the hidden grounding documents.
   • It scores the response on every rubric criterion, producing a vector of partial‑credit rewards.

3. Policy optimization

   • The policy (Llama‑3.1‑8B‑Instruct) never sees the grounding documents; it only receives the multi‑dimensional reward.
   • Training uses Group Relative Policy Optimization (GRPO), an RL algorithm that normalizes rewards across groups of trajectories to stabilize learning with noisy, multi‑criterion signals.

4. Evaluation

   • A held‑out set of rubrics measures how well the fine‑tuned model aligns with the judge’s scoring.
   • Standard reasoning benchmarks (GSM8K, MATH, GPQA) test transfer to tasks not represented in the training data.

Results & Findings

• The rubric‑grounded model consistently outperforms the base model on all four downstream reasoning tasks, despite those tasks being outside the original document corpus.
• The multi‑criterion reward provides richer learning signals, enabling the policy to correct specific weaknesses (e.g., missing steps, poor justification) rather than only learning to “get the answer right”.

Practical Implications

• More reliable fine‑tuning: Developers can define explicit rubrics for desired behaviors (e.g., safety, factuality, code style) and let an LLM judge enforce them, reducing reliance on noisy human feedback.
• Partial‑credit learning: By rewarding intermediate reasoning steps, models become better at chain‑of‑thought generation, which is valuable for debugging, education, and complex decision‑support systems.
• Domain‑specific expertise: The framework can ingest proprietary documentation (API specs, internal policies) to produce rubrics that guide a model without exposing the raw documents to the model itself—useful for privacy‑sensitive industries.
• Transferable reasoning: Training on structured rewards derived from one domain can improve performance on unrelated reasoning tasks, suggesting a cost‑effective way to boost general problem‑solving abilities without massive multi‑task datasets.

Limitations & Future Work

• Judge dependency: The quality of the reward hinges on the frozen LLM judge; biases or errors in the judge propagate to the policy.
• Rubric design overhead: Automatically extracting meaningful criteria from arbitrary corpora remains non‑trivial and may require domain expertise.
• Scalability: Experiments were limited to an 8‑B parameter model; it is unclear how the approach scales to larger models or more complex multi‑modal tasks.
• Future directions: The authors suggest exploring adaptive rubric weighting, multi‑judge ensembles for robustness, and applying the method to code generation, dialogue safety, and multimodal reasoning.

Authors

• Manish Bhattarai
• Ismael Boureima
• Nishath Rajiv Ranasinghe
• Scott Pakin
• Dan O’Malley

Paper Information

• arXiv ID: 2605.08061v1
• Categories: cs.AI
• Published: May 8, 2026
• PDF: Download PDF