[Paper] Reward Modeling for Multi-Agent Orchestration
Source: arXiv - 2606.13598v1
Overview
Multi-Agent Systems (MAS) built on Large Language Models (LLMs) require effective orchestration to coordinate specialized agents, yet training such orchestrators is hindered by limited supervision and high computational cost. We propose Orchestration Reward Modeling (OrchRM), a self-supervised framework for evaluating orchestration quality without human annotations. OrchRM leverages intermediate artifacts from multi-agent executions to construct win-lose pairs for Bradley-Terry reward model training. Unlike existing MAS test-time scaling and orchestrator training frameworks that rely on costly sub-agent rollouts, OrchRM operates directly at the orchestration level, enabling efficient and high-performing reward-guided orchestrator training and MAS test-time scaling. OrchRM improves training efficiency by up to 10x in token usage while improving MAS test-time scaling performance by up to 8% in accuracy. These gains consistently transfer across multiple domains, including mathematical reasoning, web-based question answering, and multi-hop reasoning, demonstrating orchestration-level reward modeling as a scalable direction for robust multi-agent orchestration. Code will be available at https://github.com/Wang-ML-Lab/OrchRM.
Key Contributions
This paper presents research in the following areas:
- cs.AI
- cs.CL
- cs.LG
- cs.MA
Methodology
Please refer to the full paper for detailed methodology.
Practical Implications
This research contributes to the advancement of cs.AI.
Authors
- King Yeung Tsang
- Zihao Zhao
- Vishal Venkataramani
- Haizhou Shi
- Zixuan Ke
- Semih Yavuz
- Shafiq Joty
- Hao Wang
Paper Information
- arXiv ID: 2606.13598v1
- Categories: cs.AI, cs.CL, cs.LG, cs.MA
- Published: June 11, 2026
- PDF: Download PDF