[Paper] CM2: Reinforcement Learning with Checklist Rewards for Multi-Turn and Multi-Step Agentic Tool Use

Source: arXiv - 2602.12268v1

Overview

The paper introduces CM2, a reinforcement‑learning (RL) framework that teaches large language model (LLM) agents to handle multi‑turn conversations while dynamically invoking external tools (e.g., search APIs, calculators). Instead of relying on hard‑to‑obtain “verifiable” rewards (did the agent get the exact right answer?), CM2 uses checklist rewards—a set of binary criteria that can be automatically judged. This makes RL feasible at scale, even when the ultimate task is open‑ended or subjective.

Key Contributions

• Checklist‑based reward design: Decomposes each turn’s intended behavior into fine‑grained, binary criteria with explicit evidence, turning vague success signals into stable classification decisions.
• Sparse‑reward, dense‑evaluation strategy: Rewards are given only when a turn is completed, but the evaluation covers many criteria, preserving learning signal richness without destabilizing training.
• LLM‑simulated tool environment: Provides a cheap, scalable sandbox where tool calls are emulated by a language model, eliminating the need to maintain a large fleet of real external services.
• Empirical gains over supervised fine‑tuning (SFT): Starting from an 8‑billion‑parameter base model, CM2 yields +8 τ‑Bench, +10 BFCL‑V4, and +12 ToolSandbox points compared with the SFT baseline.
• Open‑source release: Full code and data are publicly available, encouraging reproducibility and community extensions.

Methodology

1. Turn‑level decomposition – For each conversational turn, the authors define a checklist (e.g., “Did the agent correctly parse the user intent?”, “Was the appropriate tool selected?”, “Is the tool’s output cited in the response?”). Each item is a binary label that can be automatically verified using simple heuristics or a lightweight classifier.
2. Reward assignment – When a turn finishes, the checklist is evaluated. If all required items are satisfied, the agent receives a positive reward; otherwise, it gets zero (or a small penalty). This sparse‑reward scheme keeps the RL signal stable while the checklist itself provides dense feedback for the policy gradient.
3. Tool simulation – Instead of calling real APIs, a separate LLM acts as a “tool oracle” that takes the tool request and returns a plausible output. This simulated environment is fast, deterministic, and cheap to run at scale.
4. RL training – The policy (the agent) is optimized with Proximal Policy Optimization (PPO). The loss combines the usual PPO objective with a KL‑penalty to keep the policy close to the SFT initialization, preventing catastrophic drift.
5. Curriculum & data – The authors construct an 8k‑example RL dataset covering diverse tasks (search, calculation, code execution). The dataset is split into episodes of 2‑5 turns, encouraging the agent to plan across multiple steps.

Results & Findings

• Consistent improvements across all three benchmarks, despite the same model size and training data.
• Parity with open‑source baselines that use handcrafted reward models or real tool APIs, showing that checklist rewards can replace expensive judging models.
• Ablation studies reveal that dense checklist evaluation is crucial; removing it drops performance by ~5 points, while using dense rewards (reward every step) destabilizes training.
• Generalization: Agents trained with CM2 handle unseen tool combinations better than SFT, suggesting the checklist encourages robust reasoning patterns rather than memorization.

Practical Implications

• Rapid prototyping of tool‑augmented assistants – Developers can now train agents to orchestrate APIs without building a full‑fidelity sandbox; the LLM‑simulated tool layer is enough for early‑stage experimentation.
• Cost‑effective RL pipelines – Checklist rewards eliminate the need for expensive human‑in‑the‑loop labeling or custom reward models, reducing the compute budget for RL fine‑tuning.
• Better safety & interpretability – Because each checklist item is explicit (e.g., “Did the agent cite the tool output?”), failures can be diagnosed quickly, aiding debugging and compliance audits.
• Plug‑and‑play for existing LLM stacks – The method works on any base model that can be fine‑tuned with PPO, making it applicable to open‑source LLMs (LLaMA, Mistral, etc.) and even proprietary offerings via adapters.
• Scalable multi‑step workflows – Industries that require chained tool usage—financial analysis (fetch market data → compute risk metrics → generate report), DevOps (query logs → run diagnostics → suggest fixes)—can adopt CM2 to improve autonomous agent reliability.

Limitations & Future Work

• Checklist design overhead – Crafting high‑quality checklists for new domains still requires domain expertise; automating checklist generation remains an open problem.
• Simulation fidelity – The LLM‑based tool oracle may produce unrealistic outputs for edge cases, potentially leading to over‑optimistic policies when deployed against real APIs.
• Sparse reward sparsity – While stabilizing training, the binary “all‑or‑nothing” reward can penalize partially correct behavior; future work could explore graded rewards or curriculum‑based relaxation.
• Scalability to larger models – Experiments focus on an 8B model; it is unclear how the approach scales to 70B+ models where policy drift and KL‑penalties behave differently.
• User‑centric evaluation – Benchmarks measure task success, but real‑world user satisfaction (e.g., perceived helpfulness, trust) is not directly assessed; integrating human‑in‑the‑loop feedback would strengthen the framework.

CM2 demonstrates that with the right reward abstraction—checklist‑style binary criteria—reinforcement learning can finally be applied to the messy, multi‑turn, tool‑using agents that developers need today. By lowering the engineering barrier and delivering solid performance gains, it opens a practical path toward more autonomous, reliable AI assistants.

Authors

• Zhen Zhang
• Kaiqiang Song
• Xun Wang
• Yebowen Hu
• Weixiang Yan
• Chenyang Zhao
• Henry Peng Zou
• Haoyun Deng
• Sathish Reddy Indurthi
• Shujian Liu
• Simin Ma
• Xiaoyang Wang
• Xin Eric Wang
• Song Wang

Paper Information

• arXiv ID: 2602.12268v1
• Categories: cs.AI
• Published: February 12, 2026
• PDF: Download PDF