TokenRouter: Efficient Serving System for Token-Level LLM Routing

2026-10-08

AI Summary

This study addresses the limitations of existing systems that lack fine-grained, token-level large language model (LLM) routing support, which leads to pacing mismatches, batching latency, and high development complexity. To overcome these challenges, this work proposes a request-centric programming and model-centric execution architecture that decouples logical descriptions from physical execution by distributing requests through asynchronous sub-servers. Furthermore, it designs a delay-batching scheduler grounded in a mathematical throughput model, transcending the constraints of single-model assumptions. The primary contribution is a developer-friendly inference serving framework enabling token-level routing. Experimental evaluations demonstrate that the proposed system achieves decoding throughput improvements ranging from 2.01× to 64.15× over existing baselines across diverse scenarios, significantly optimizing LLM serving efficiency.

Abstract

Large language model (LLM) routing distributes inference work across different models, advancing the cost-quality Pareto frontier of LLM serving. While coarse-grained routing at the session or query level has been widely adopted in production systems, recent algorithmic work shows that fine-grained token-level routing can yield substantial efficiency and quality gains. However, efficiently serving token-level routed inference poses significant challenges to existing systems. Built on single-LLM assumptions, current systems suffer from severe step desynchronization and frequent batch admission delays under token-level routing, and they also impose high implementation complexity on developers. To address these challenges, we design TokenRouter, an efficient and developer-friendly serving system for token-level routed LLM inference. TokenRouter follows the principle of request-centric programming, model-centric execution: developers describe routing logic from the perspective of a single request, while the runtime launches a subserver for each LLM and dispatches requests asynchronously. Each subserver employs a delayed-batching scheduler, whose optimal hyperparameters are derived from a mathematical throughput model of the system. Across diverse routing algorithms, workloads, and model pairs, TokenRouter achieves 2.01-64.15x higher decoding throughput than existing systems, substantially advancing the serving efficiency of token-level LLM routing. Our code is available at https://github.com/thu-nics/TokenRouter.

Problem

• Token-level routing
• LLM serving
• Step desynchronization
• Batch admission delay
• Inference efficiency

Innovation

• Token-level routing
• LLM serving system
• Delayed-batching scheduler
• Throughput model
• Request-centric programming