BALANCE: Hybrid Autoregressive-Speculative LLM Inference in Wireless Edge Networks

2026-08-06

AI Summary

This work addresses the challenge of simultaneously meeting heterogeneous user demands for low latency and limited memory in wireless edge networks, where conventional single-decoding strategies fall short. To this end, the authors propose BALANCE, a novel framework that co-deploys large and small language models at the edge and dynamically assigns users to either autoregressive or speculative decoding modes. BALANCE jointly optimizes user scheduling and computational resource allocation to maximize task throughput, integrating a hybrid inference architecture with an NP-hard resource scheduling problem. The framework features an efficient algorithm with a constant approximation ratio. Experimental results demonstrate that BALANCE significantly outperforms existing single-decoding approaches, achieving substantially higher system throughput while satisfying stringent user-specific latency and memory constraints.

Abstract

Edge inference is a promising paradigm to provide large language model (LLM) inference services in next-generation mobile networks. LLM inference mainly relies on two approaches: Autoregressive decoding (AD) generates output tokens sequentially, resulting in long latency; Speculative decoding (SD) accelerates inference by using a small language model (SLM) to generate multiple draft tokens for LLM verification, but incurs extra memory costs. Due to this latency-memory tradeoff, neither approach alone can efficiently serve users with heterogeneous demands under limited edge computing resources. To address this challenge, we propose a hybrid autoregressive-speculative inference (BALANCE) framework for edge LLM inference. In BALANCE, an edge server hosts both an SLM and an LLM, assigns each user to AD or SD, and performs the two modes simultaneously. To maximize the number of served users, we formulate a task throughput maximization problem to jointly determine user scheduling and computing resource allocation between AD and SD under user latency requirements and server memory constraints. Since the problem is NP-hard, we develop a polynomial-time algorithm that transforms the original problem into two sub-problems and obtains a sub-optimal solution with a constant approximation guarantee. Experiments demonstrate that BALANCE consistently outperforms conventional AD and SD and significantly improves task throughput.

Problem

• edge inference
• large language model
• latency-memory tradeoff
• heterogeneous user demands
• resource-constrained edge computing

Innovation

• hybrid inference
• speculative decoding
• edge computing
• resource allocation
• LLM acceleration