Reward Modeling for Multi-Agent Orchestration

2026-06-11

AI Summary

This work addresses the challenges of training large language model (LLM)-based coordinators in multi-agent systems, where supervision signals are scarce and computational costs are prohibitive. The authors propose OrchRM, a novel framework that enables self-supervised reward modeling without requiring rollouts from individual agents, relying solely on signals from the coordination layer. By constructing win-loss pairs from intermediate artifacts generated during multi-agent execution and integrating them with a Bradley-Terry reward model and test-time scaling techniques, OrchRM efficiently trains coordinators without human annotations. The approach substantially improves training efficiency and cross-domain generalization, reducing token consumption by up to 10× and boosting test-time accuracy by as much as 8%, consistently outperforming baselines across diverse tasks including mathematical reasoning, web-based question answering, and multi-hop reasoning.

Abstract

Multi-Agent Systems (MAS) built on Large Language Models (LLMs) require effective orchestration to coordinate specialized agents, yet training such orchestrators is hindered by limited supervision and high computational cost. We propose Orchestration Reward Modeling (OrchRM), a self-supervised framework for evaluating orchestration quality without human annotations. OrchRM leverages intermediate artifacts from multi-agent executions to construct win-lose pairs for Bradley-Terry reward model training. Unlike existing MAS test-time scaling and orchestrator training frameworks that rely on costly sub-agent rollouts, OrchRM operates directly at the orchestration level, enabling efficient and high-performing reward-guided orchestrator training and MAS test-time scaling. OrchRM improves training efficiency by up to 10x in token usage while improving MAS test-time scaling performance by up to 8% in accuracy. These gains consistently transfer across multiple domains, including mathematical reasoning, web-based question answering, and multi-hop reasoning, demonstrating orchestration-level reward modeling as a scalable direction for robust multi-agent orchestration. Code will be available at https://github.com/Wang-ML-Lab/OrchRM.

Problem

• Multi-Agent Systems
• Reward Modeling
• Orchestration
• Large Language Models
• Self-supervised Learning

Innovation

• Orchestration Reward Modeling
• Multi-Agent Systems
• Self-supervised Learning
• Bradley-Terry Model
• Test-time Scaling