[Paper] Data-Efficient Hierarchical Goal-Conditioned Reinforcement Learning via Normalizing Flows

Source: arXiv - 2602.11142v1

Overview

The paper introduces Normalizing‑Flow Hierarchical Implicit Q‑Learning (NF‑HIQL), a new way to train hierarchical goal‑conditioned reinforcement learning agents that works well even when only a small amount of data is available. By swapping the usual simple Gaussian policies for expressive normalizing‑flow policies at both the high‑level (goal proposer) and low‑level (skill executor), the authors achieve better performance on long‑horizon tasks while keeping the learning process stable and tractable.

Key Contributions

• Flow‑based policies for hierarchy: Replaces unimodal Gaussian action distributions with normalizing‑flow models (RealNVP) at both hierarchy levels, enabling multimodal and highly expressive behavior.
• Tractable likelihood & sampling: The flow architecture provides exact log‑likelihoods and efficient sampling, which are essential for off‑policy Q‑learning updates.
• Theoretical guarantees: Derives explicit KL‑divergence bounds for RealNVP policies and PAC‑style sample‑efficiency results, showing that the richer policy class does not sacrifice stability.
• Data‑efficient learning: Demonstrates superior performance in offline or low‑data regimes compared to prior goal‑conditioned and hierarchical baselines.
• Extensive empirical validation: Benchmarks on locomotion, ball‑dribbling, and multi‑step manipulation tasks from OGBench, with consistent gains across environments.

Methodology

1. Hierarchical Goal‑Conditioned RL (H‑GCRL) setup – A high‑level policy proposes intermediate subgoals, and a low‑level policy tries to achieve each subgoal using primitive actions.
2. Implicit Q‑Learning backbone – The authors build on Implicit Q‑Learning (IQL), an off‑policy algorithm that learns a value function and derives a policy by minimizing a weighted Bellman error, avoiding explicit policy gradients.
3. Normalizing‑Flow policies – Both high‑ and low‑level policies are parameterized as RealNVP flows: a series of invertible transformations that map a simple base distribution (e.g., standard Gaussian) to a complex target distribution. This gives:
   • Exact log‑probability for the policy (needed for the IQL loss).
   • Ability to capture multimodal action distributions (e.g., multiple ways to reach a subgoal).
4. Training loop –
   • Sample transitions from a replay buffer (offline or online).
   • Update the Q‑function with standard temporal‑difference targets.
   • Update the flow policies by minimizing the IQL objective, which includes a KL‑regularizer that now has a closed‑form expression thanks to the flow’s tractable Jacobian.
5. Theoretical analysis – Proves that the KL divergence between the learned flow policy and the optimal policy can be bounded, and provides a PAC‑style bound on the number of samples needed to achieve a target performance level.

Results & Findings

• Robustness to data scarcity: When training with only 10 % of the full dataset, NF‑HIQL retains >80 % of its full‑data performance, whereas baselines drop below 50 %.
• Multimodal behavior: Visualizations of low‑level action distributions show distinct modes corresponding to alternative strategies (e.g., going around an obstacle vs. jumping over it).
• Stability: Training curves exhibit lower variance and fewer catastrophic drops compared with Gaussian‑policy hierarchies, confirming the theoretical stability claims.

Practical Implications

• Offline RL for robotics: Engineers can now train hierarchical policies from limited logged data (e.g., tele‑operated demonstrations) without needing massive on‑policy rollouts.
• Complex task decomposition: The expressive flow policies make it easier to encode multiple viable sub‑strategies, which is valuable for tasks like autonomous driving where several safe maneuvers exist.
• Plug‑and‑play with existing pipelines: NF‑HIQL is built on top of standard IQL codebases; swapping the policy network for a RealNVP module is the only change required, lowering adoption friction.
• Better exploration in simulation‑to‑real transfer: Multimodal low‑level policies can cover a richer set of behaviors, increasing the chance that a simulated policy will find a feasible real‑world execution path.

Limitations & Future Work

• Computational overhead: Normalizing‑flow networks are heavier than simple Gaussians, leading to ~1.5× longer training time and higher memory usage.
• Scalability to very high‑dimensional action spaces: While RealNVP scales reasonably, extremely high‑dimensional robotics (e.g., dexterous hands) may require more sophisticated flow architectures or dimensionality reduction.
• Limited offline benchmark diversity: Experiments focus on continuous control; extending to discrete or mixed action spaces remains an open question.
• Future directions: The authors suggest exploring flow‑based policies for the high‑level planner only (to reduce cost), integrating learned flows with model‑based RL for further data efficiency, and applying NF‑HIQL to real‑world robotic platforms.

Authors

• Shaswat Garg
• Matin Moezzi
• Brandon Da Silva

Paper Information

• arXiv ID: 2602.11142v1
• Categories: cs.RO, cs.AI, cs.LG
• Published: February 11, 2026
• PDF: Download PDF