PROTEUS: SLA-Aware Routing via Lagrangian RL for Multi-LLM Serving Systems

2026-01-27

AI Summary

This work proposes PROTEUS, the first SLA-aware dynamic routing framework for large language models (LLMs) that directly accepts a user-specified accuracy target τ as input. Existing LLM routing systems struggle to meet accuracy service-level agreements (SLAs) at runtime, relying on manual offline tuning and offering no formal guarantees. In contrast, PROTEUS internalizes SLA constraints into routing decisions via Lagrangian dual reinforcement learning, enabling it to satisfy any accuracy target across the full spectrum without model retraining. Experimental results on RouterBench and SPROUT demonstrate that PROTEUS achieves near-perfect adherence to target accuracy (≈100% success rate), exhibits strong correlation between target and actual accuracy (0.97–0.98), matches the performance of an oracle routing policy, and reduces inference costs by up to 89.8%.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Natural Language Processing: Safety and Robustness
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Production LLM deployments serve diverse workloads where cost and quality requirements vary by customer tier, time of day, and query criticality. Model serving systems accept latency SLOs directly. LLM routers do not. They force operators to tune parameters offline and guess what accuracy might result. The relationship between parameters and outcomes is indirect, non-monotonic, and dataset-dependent. Operators need to specify accuracy targets, not infer them from opaque settings. We present PROTEUS (Polymorphic Router for Operational Target Enforcement with Unified SLA), a router that accepts accuracy targets tau as runtime input. PROTEUS uses Lagrangian dual control. A learned dual variable lambda tracks constraint violations during training and conditions the policy network. This lets the router translate specified tau values into routing decisions that satisfy them. A single trained model serves the full accuracy spectrum without retraining.We evaluate on RouterBench (11 models, 405K queries) and SPROUT (14 models, 45K queries). PROTEUS achieves consistent floor compliance where accuracy meets or exceeds tau. The target-response correlation reaches 0.97 to 0.98. The closest baseline, OmniRouter, meets floors only 22% of the time despite also using Lagrangian optimization. PROTEUS operates across tau in [0.85, 0.95] from a single model. On RouterBench it achieves 90.1% accuracy, within 1.3% of oracle. On SPROUT it achieves 94.0% accuracy, within 4.6% of oracle. Cost savings reach 89.8% versus the best fixed model.

Problem

• LLM serving
• SLA-aware routing
• accuracy targets
• multi-LLM systems
• quality-of-service

Innovation

• Lagrangian Reinforcement Learning
• SLA-aware routing
• Multi-LLM serving
• Accuracy target enforcement
• Dual control