Quality-of-Service Aware LLM Routing for Edge Computing with Multiple Experts

2025-07-31

AI Summary

Ensuring Quality-of-Service (QoS) for multi-expert large language models (LLMs) in edge computing—specifically, achieving low-latency, high-quality request routing amid service heterogeneity, inter-request interference, and dynamic workloads—remains a critical challenge. Method: We propose a deep reinforcement learning–driven intelligent routing framework that jointly incorporates dynamic state abstraction and a heterogeneous graph attention network to efficiently encode global system state; further, we design an action impact estimator and a QoS-aware reward function to explicitly model service disparities and interference effects. Results: Experiments under both Poisson and real-world workloads demonstrate that our approach significantly reduces latency violation rates, improves average QoS, and enhances resource utilization efficiency compared to state-of-the-art baselines.

Technology Category

Planning, Routing, and Scheduling: Planning with Language ModelsMachine Learning: Learning on the Edge & Model CompressionSearch and Optimization: Learning to Search

Application Category

Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphsEconomics, Online Markets and Human Computation: Cost models of using LLMs in production systemsSearch and Retrieval-Augmented AI: Large language models for search

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities, leading to a significant increase in user demand for LLM services. However, cloud-based LLM services often suffer from high latency, unstable responsiveness, and privacy concerns. Therefore, multiple LLMs are usually deployed at the network edge to boost real-time responsiveness and protect data privacy, particularly for many emerging smart mobile and IoT applications. Given the varying response quality and latency of LLM services, a critical issue is how to route user requests from mobile and IoT devices to an appropriate LLM service (i.e., edge LLM expert) to ensure acceptable quality-of-service (QoS). Existing routing algorithms fail to simultaneously address the heterogeneity of LLM services, the interference among requests, and the dynamic workloads necessary for maintaining long-term stable QoS. To meet these challenges, in this paper we propose a novel deep reinforcement learning (DRL)-based QoS-aware LLM routing framework for sustained high-quality LLM services. Due to the dynamic nature of the global state, we propose a dynamic state abstraction technique to compactly represent global state features with a heterogeneous graph attention network (HAN). Additionally, we introduce an action impact estimator and a tailored reward function to guide the DRL agent in maximizing QoS and preventing latency violations. Extensive experiments on both Poisson and real-world workloads demonstrate that our proposed algorithm significantly improves average QoS and computing resource efficiency compared to existing baselines.

Problem

• Route user requests to optimal edge LLM for QoS
• Address heterogeneity and dynamic workloads in LLM routing
• Ensure high QoS with DRL and dynamic state abstraction

Innovation

• DRL-based QoS-aware LLM routing framework
• Dynamic state abstraction with HAN
• Action impact estimator and reward function