[Paper] Decoupled Q-Chunking

Source: arXiv - 2512.10926v1

Overview

Temporal‑difference (TD) reinforcement learning is great at learning value functions quickly, but the bootstrapping it relies on can introduce a nasty “bootstrapping bias” that compounds errors over many steps. Recent work tried to fix this by using chunked critics—value estimators that look ahead over short sequences of actions instead of a single step. The catch? Extracting a usable policy from such critics forces the policy to output whole action chunks open‑loop, which hurts reactivity and becomes hard to train as the chunk length grows.

The new paper “Decoupled Q‑Chunking” proposes a simple yet powerful twist: decouple the chunk length used by the critic from the chunk length used by the policy. By doing so, the algorithm keeps the multi‑step learning benefits while letting the policy remain responsive and easier to train.

Key Contributions

• Decoupled chunk lengths – Introduces a framework where the critic evaluates long action chunks (e.g., 10‑step sequences) but the policy only needs to output shorter chunks (e.g., 2‑step sequences).
• Optimistic partial‑chunk backup – Derives a distilled critic for partial chunks by backing up optimistically from the original chunked critic, approximating the best possible completion of a partial sequence.
• Algorithmic pipeline – Provides a concrete training loop that alternates between (1) learning the long‑horizon chunked critic, (2) constructing the distilled partial‑chunk critic, and (3) updating the short‑horizon policy against it.
• Empirical validation – Demonstrates consistent gains on challenging offline, goal‑conditioned, long‑horizon benchmarks (e.g., robotic manipulation and navigation tasks).
• Open‑source implementation – Releases code (github.com/ColinQiyangLi/dqc) to facilitate reproducibility and downstream adoption.

Methodology

1. Chunked Critic Learning

   • The critic (Q_{\text{chunk}}(s, a_{0:k-1})) predicts the return of a k‑step open‑loop action sequence (the “chunk”).
   • Standard TD updates are applied, but the target now spans k steps, reducing the number of bootstrapping operations and thus the accumulated bias.

2. Distilling a Partial‑Chunk Critic

   • For a shorter policy chunk length m (where m < k), the authors construct a partial‑chunk value:
     [ \tilde{Q}(s, a_{0:m-1}) = \max_{a_{m:k-1}} Q_{\text{chunk}}(s, a_{0:k-1}) ]
   • Since enumerating all completions is infeasible, they approximate the max by optimistic backup: run a short rollout using the current policy for the remaining k‑m steps and add the resulting value estimate.

3. Policy Optimization

   • The policy (\pi_{\theta}) now outputs m‑step action chunks.
   • It is trained to maximize the distilled partial‑chunk critic (\tilde{Q}) via standard policy‑gradient or actor‑critic updates.
   • Because the policy only needs to plan a few steps ahead, it can react to new observations between chunks, preserving reactivity.

4. Training Loop

   • Step A: Update the long‑horizon chunked critic with TD targets.
   • Step B: Build the distilled partial‑chunk critic from the updated chunked critic.
   • Step C: Update the short‑horizon policy against the distilled critic.
   • Repeat until convergence.

The whole pipeline is compatible with offline datasets (no environment interaction needed) and can be plugged into existing RL libraries with minimal changes.

Results & Findings

• Bias reduction: The multi‑step backup dramatically lowered TD error propagation, especially noticeable in the later stages of long episodes.
• Policy reactivity: Shorter policy chunks allowed the agent to adapt mid‑trajectory, leading to higher goal‑reachability in environments with dynamic obstacles.
• Scalability: Training time grew only modestly with longer critic chunks because the policy updates remained cheap (short‑horizon).

Overall, Decoupled Q‑Chunking consistently outperformed both classic TD methods and prior chunked‑critic approaches across all tested domains.

Practical Implications

In short, Decoupled Q‑Chunking offers a pragmatic recipe: keep the learning horizon long to get better credit assignment, but keep the execution horizon short to stay responsive. This matches how many production systems are architected (high‑level planner + low‑level controller), making the method a natural fit for real‑world pipelines.

Limitations & Future Work

• Optimistic backup approximation – The distilled partial‑chunk critic relies on a heuristic rollout; if the policy used for the rollout is poor, the approximation can be biased.
• Offline‑only evaluation – Experiments were limited to offline datasets; extending to online RL (where the policy can influence data collection) remains an open question.
• Fixed chunk lengths – The paper uses static lengths for critic and policy chunks. Adaptive or state‑dependent chunk sizing could further improve efficiency.
• Scalability to very high‑dimensional actions – While the method reduces the burden on the policy, learning a long‑horizon chunked critic in extremely high‑dim spaces (e.g., raw pixel actions) may still be challenging.

Future research directions suggested by the authors include: (1) learning the optimal chunk lengths jointly, (2) integrating model‑based rollouts for a more accurate partial‑chunk backup, and (3) applying the framework to online, exploration‑driven settings.

Authors

• Qiyang Li
• Seohong Park
• Sergey Levine

Paper Information

• arXiv ID: 2512.10926v1
• Categories: cs.LG, cs.AI, cs.RO, stat.ML
• Published: December 11, 2025
• PDF: Download PDF