TRACE-ROUTER: Task-Consistent and Adaptive Online Routing for Agentic AI

2026-07-24

AI Summary

This work addresses the limitations of existing large language model (LLM) routing mechanisms, which make independent decisions at each invocation and thus struggle to support long-horizon agent tasks or properly attribute delayed task-level feedback to specific routing choices. The authors propose a task-level online routing framework that, at task initiation, uses a contextual bandit to assign a single LLM for the entire task duration, thereby ensuring consistent model usage. By leveraging the final task reward, the framework jointly optimizes accuracy and latency without requiring explicit estimation of task complexity. This approach uniquely aligns routing decisions with task-level supervision signals, enabling adaptive yet consistent routing. Evaluated on three agent benchmarks, it significantly improves the accuracy–latency trade-off—yielding 7–8 absolute percentage points higher accuracy than latency-matched interpolation on tau2-Bench and surpassing the strongest single-model baseline on Terminal-Bench by 7.1 accuracy points while reducing latency by 36%.

Abstract

Routing to select large language models (LLMs) with different cost-quality trade-offs has become a fundamental deployment feature of enterprise AI. Existing routers, primarily make independent routing decisions for each LLM call. However, agentic applications execute as long-horizon workflows whose quality is determined only by a delayed, task-level outcome. This mismatch prevents per-call routers from correctly attributing feedback to individual routing decisions. Towards mitigating this, we present TRACE-Router, a task-level routing framework that aligns routing with the unit of supervision. TRACE-Router assigns each task to a model once at admission using a contextual bandit, pins all subsequent LLM calls to the selected backend, and updates its policy using the task's terminal reward, jointly accounting for accuracy and latency. By leveraging delayed task feedback, TRACE-Router learns routing policies that adapt to the workload while avoiding explicit task-complexity estimation. Across three agentic benchmarks, TRACE-Router consistently improves the accuracy-latency trade-off, achieving non-dominated Pareto frontier points. On tau2-Bench, it outperforms latency-matched interpolation between individual models by 7-8 accuracy points, while on Terminal-Bench it achieves 7.1 higher accuracy points than the strongest single model baseline with 36% lower latency.

Problem

• agentic AI
• online routing
• task-level feedback
• LLM selection
• accuracy-latency trade-off

Innovation

• task-level routing
• contextual bandit
• agentic AI
• delayed feedback
• Pareto frontier