Multi-SPIN: Multi-Access Speculative Inference for Cooperative Token Generation at the Edge

2026-06-03

AI Summary

This work addresses the challenges of computational load imbalance between devices and servers, as well as user heterogeneity, in multi-user edge computing systems. To this end, the authors propose Multi-SPIN, a novel architecture wherein lightweight on-device models generate draft tokens for upload, and an edge server performs parallel batched verification. This is the first effort to extend speculative inference to multi-user edge scenarios. The framework introduces both homogeneous and heterogeneous draft-length control strategies and derives a closed-form, efficient algorithm through joint optimization of draft length and bandwidth allocation, thereby overcoming the limitations of conventional synchronous batching. Evaluated with co-deployed Llama-2 and Qwen-1.5 (noting that “Qwen3.5” in the original appears to be a typographical error), the system achieves up to an 88% improvement in token throughput over baselines that disregard user heterogeneity across diverse tasks.

Abstract

Speculative inference (SPIN) was originally developed as an efficient architecture to accelerate Large Language Models (LLMs). In this work, we propose its distributed deployment to enable cooperative token generation in a multiuser edge system; its advantage is to effectively balance computational loads between resource-constrained devices and servers. The resulting architecture, termed Multi-access SPIN (Multi-SPIN), utilizes on-device small language models to generate and upload candidate token drafts, while an edge server operates the LLM to verify them in parallel batches. Given the severe heterogeneity in users' computation and communication capabilities, the draft length emerges as a critical control variable that influences node-level computation loads and multi-access latency, thereby governing the sum token goodput. Consequently, considering frequency-division multiple access, we investigate the problem of multi-access draft control, a joint optimization of draft-length control and bandwidth allocation to maximize sum token goodput. We examine two cases: (1) homogeneous draft lengths across users to facilitate server-side batching, and (2) heterogeneous draft lengths to introduce a new dimension for goodput enhancement. By developing decomposition methods, we reduce these complex optimizations into tractable sub-problems, which allow efficient draft control algorithms to be derived in closed form. Our analysis shows that the optimal bandwidth allocation compensates users with weaker computation-and-communication capabilities in the homogeneous case due to the batching synchronization requirements, whereas its heterogeneous-case counterpart rewards users with higher acceptance rates by relaxing such requirements. Experiments using Llama-2 and Qwen3.5 model pairs across diverse tasks demonstrate that Multi-SPIN improves goodput by up to 88% over heterogeneity-agnostic baselines.

Problem

• speculative inference
• edge computing
• token goodput
• draft-length control
• bandwidth allocation

Innovation

• Speculative Inference
• Edge Computing
• Token Goodput
• Multi-access Optimization
• Distributed LLM