LMEdge: QoS-Aware LLM Inference Orchestration on Edge Clusters

2026-07-19

AI Summary

This work addresses the multidimensional QoS scheduling challenge of large language model (LLM) inference in heterogeneous, resource-constrained edge environments. It presents the first QoS-aware scheduling framework that jointly optimizes model configuration—including model family, scale, and quantization level—and execution placement. The problem is formulated as a constrained binary integer linear program, supported by five lightweight predictive models that estimate latency, accuracy, resource consumption, and response size in real time. An efficient heuristic algorithm is designed to enable scalable solutions. Evaluation on a 57-node Kubernetes-based edge testbed demonstrates that the proposed approach significantly reduces response time compared to baseline methods while maintaining model accuracy, improving resource utilization, and increasing service success rate.

Abstract

Large language model (LLM) services increasingly operate on edge infrastructure, enabling low-latency and privacy-preserving AI services. However, efficiently serving LLM requests across heterogeneous and resource-constrained edge devices require orchestration mechanisms that jointly determine model configuration (family, size, and quantization level) and execution placement while satisfying user- and system-level quality of service (QoS) requirements. This paper introduces LMEdge, a QoS-aware orchestration service that dynamically makes these decisions across heterogeneous edge devices. We formulate the problem as a binary integer linear programming (BILP) optimization that minimizes response time under accuracy, network, and resource constraints. To enable scalable online scheduling, we employ five lightweight machine learning (ML) models to predict query-specific latency, accuracy, resource usage, and response size for each model-size-quantization-device combination, and design a lightweight heuristic that approximates the BILP solution. We collect a comprehensive benchmarking dataset of over 59000 rows to train models and support reproducibility. Evaluation on a Kubernetes-based edge testbed with 57 instances and diverse query categories shows that LMEdge reduces latency, preserves accuracy, improves resource utilization, and increases serving ratio compared to two baselines.

Problem

• LLM inference
• edge computing
• QoS-aware orchestration
• resource-constrained devices
• heterogeneous edge clusters

Innovation

• LLM inference orchestration
• QoS-aware scheduling
• edge computing
• binary integer linear programming
• lightweight ML prediction