Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey

2026-02-23

AI Summary

Static deployment of large language models (LLMs) struggles to dynamically select the optimal model based on query complexity and domain, leading to an imbalance between performance and cost. This work proposes a three-dimensional framework—centered on decision timing, information sources, and computation mechanisms—to systematically analyze dynamic routing and cascading strategies across multiple LLMs. It introduces the first taxonomy of routing paradigms spanning independently trained LLMs and integrates diverse technical approaches, including query difficulty estimation, human preference modeling, uncertainty quantification, reinforcement learning, and multimodal fusion. Experimental results demonstrate that well-designed routing systems can surpass the strongest individual model, achieving a superior trade-off between inference efficiency and performance, while also highlighting critical challenges such as generalization.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The rapid growth of large language models (LLMs) with diverse capabilities, costs, and domains has created a critical need for intelligent model selection at inference time. While smaller models suffice for routine queries, complex tasks demand more capable models. However, static model deployment does not account for the complexity and domain of incoming queries, leading to suboptimal performance and increased costs. Dynamic routing systems that adaptively select models based on query characteristics have emerged as a solution to this challenge. We provide a systematic analysis of state-of-the-art multi-LLM routing and cascading approaches. In contrast to mixture-of-experts architectures, which route within a single model, we study routing across multiple independently trained LLMs. We cover diverse routing paradigms, including query difficulty, human preferences, clustering, uncertainty quantification, reinforcement learning, multimodality, and cascading. For each paradigm, we analyze representative methods and examine key trade-offs. Beyond taxonomy, we introduce a conceptual framework that characterizes routing systems along three dimensions: when decisions are made, what information is used, and how they are computed. This perspective highlights that practical systems are often compositional, integrating multiple paradigms under operational constraints. Our analysis demonstrates that effective multi-LLM routing requires balancing competing objectives. Choosing the optimal routing strategy depends on deployment and computational constraints. Well-designed routing systems can outperform even the most powerful individual models by strategically leveraging specialized capabilities across models while maximizing efficiency gains. Meanwhile, open challenges remain in developing routing mechanisms that generalize across diverse architectures, modalities, and applications.

Problem

• dynamic model routing
• LLM inference
• model selection
• query complexity
• multi-LLM systems

Innovation

• dynamic model routing
• multi-LLM cascading
• query-aware inference
• adaptive model selection
• efficient LLM inference