[Paper] ARL-Tangram: Unleash the Resource Efficiency in Agentic Reinforcement Learning

Source: arXiv - 2603.13019v1

Overview

The paper introduces ARL‑Tangram, a resource‑orchestration system designed for agentic reinforcement learning (RL) workloads that power large language models (LLMs) interacting with the real world. By moving from static, over‑provisioned resource allocation to fine‑grained, action‑level sharing, the authors achieve dramatic gains in efficiency and speed for cloud‑based RL training.

Key Contributions

• Action‑level orchestration model that treats each RL “action” (e.g., code execution, reward evaluation) as an independently schedulable unit.
• Elastic scheduling algorithm that minimizes Action Completion Time (ACT) while respecting heterogeneous CPU/GPU constraints.
• Unified heterogeneous resource manager that can dynamically allocate resources across different hardware topologies (CPU clusters, GPU farms, etc.).
• Real‑world evaluation on production‑grade agentic RL tasks, showing up to 4.3× faster ACT, 1.5× higher step throughput, and 71.2 % reduction in external resource usage.
• Deployment at scale: the system now underpins the training pipeline for the MiMo series of LLMs.

Methodology

1. Action‑Level Formulation – Instead of binding an entire RL trajectory to a fixed set of resources, the authors decompose a trajectory into atomic actions (e.g., “run generated Python code”, “score with a reward model”). Each action carries its own resource profile (CPU cores, GPU memory, network bandwidth).
2. Elastic Scheduler – A custom scheduler continuously monitors the cluster state and matches pending actions to the most suitable idle resources. It solves a lightweight optimization problem that balances two goals: (a) keep ACT as low as possible, and (b) never exceed the per‑resource capacity limits.
3. Heterogeneous Resource Managers – Separate managers handle CPUs, GPUs, and mixed‑mode nodes. They expose a common API to the scheduler, abstracting away hardware‑specific quirks (e.g., GPU warm‑up latency, CPU affinity).
4. Implementation & Integration – ARL‑Tangram is built as a plug‑in for existing agentic RL frameworks. It intercepts action dispatch calls, routes them through the scheduler, and reports back completion timestamps for feedback‑driven tuning.

The overall design keeps the system transparent to RL researchers: they write their algorithms as usual, while ARL‑Tangram silently optimizes the underlying execution.

Results & Findings

The experiments span several agentic RL benchmarks (code generation, tool‑use, web‑navigation) and demonstrate that the gains hold across diverse workloads and hardware configurations.

Practical Implications

• Cost Savings for Cloud‑Based RL – By shrinking the external resource footprint, organizations can cut cloud bills dramatically, especially when scaling to billions of RL steps.
• Higher Throughput for LLM Fine‑Tuning – Faster ACT translates directly into shorter training cycles, enabling more rapid iteration on agentic capabilities (e.g., tool‑using assistants).
• Better Cluster Utilization – Elastic sharing reduces idle CPU/GPU time, allowing other services (batch jobs, inference) to co‑exist on the same hardware without manual partitioning.
• Simplified Ops – Teams no longer need to manually over‑provision per‑task clusters; the scheduler automatically balances demand, lowering operational complexity.
• Scalable Deployment – The system’s modular resource managers make it straightforward to plug in new hardware (TPUs, specialized inference chips) as they become available.

Limitations & Future Work

• Scheduling Overhead at Extreme Scale – While negligible in current experiments, the optimization step could become a bottleneck when handling millions of concurrent actions; more distributed scheduling heuristics are needed.
• Assumption of Homogeneous Action Granularity – The model works best when actions are roughly comparable in execution time; highly variable actions may still cause stragglers.
• Integration with Proprietary Cloud APIs – The current prototype targets open‑source clusters; extending support to commercial cloud providers (AWS, Azure) will require additional adapters.
• Future Directions – The authors plan to explore reinforcement‑learning‑based schedulers that learn to predict resource demand, and to incorporate energy‑aware metrics for greener RL training.

Authors

• Bangjun Xiao
• Yihao Zhao
• Xiangwei Deng
• Shihua Yu
• Yuxing Xiang
• Huaqiu Liu
• Qiying Wang
• Liang Zhao
• Hailin Zhang
• Xuanzhe Liu
• Xin Jin
• Fuli Luo

Paper Information

• arXiv ID: 2603.13019v1
• Categories: cs.DC, cs.AI, cs.LG
• Published: March 13, 2026
• PDF: Download PDF