Problem
Research questions and friction points this paper is trying to address.
speculative decoding
asymmetric networks
cloud-edge computing
uplink bottleneck
invalid dependent work
AsymSpec: Efficient Cloud-Edge Speculative Decoding over Asymmetric Networks
🤖 AI Summary
This work addresses the inefficiencies in cloud-edge collaborative speculative decoding, where constrained uplink bandwidth often leads to queuing of candidate tokens and idle verifiers, resulting in wasted edge computation or redundant processing. To overcome this, the authors propose an asymmetric verification protocol coupled with a prefix-commit pipelining mechanism: a lightweight draft model runs on the edge while a high-quality target model executes in the cloud. Efficient communication is achieved through compact uplink transmission and downlink error correction, complemented by a total variation (TV) residual–based progressive recovery strategy for optimized scheduling. This approach effectively eliminates redundant computation caused by over-speculation, achieving output token throughput 2.82–28.03 times higher than the strongest baseline across diverse models, workloads, and network conditions.
📝 Abstract
Cloud-edge speculative decoding places a lightweight draft model at an edge gateway and a higher-quality target model in the cloud, but inserts communication into every speculative block. Under a constrained uplink, candidate messages may queue while the verifier is idle. Stop-and-wait scheduling leaves edge compute underutilized; optimistic same-request runahead can waste work when a rejection or an unexpected bonus token invalidates dependent drafts. We present AsymSpec, which addresses uplink-gated verification and invalid dependent work with two corresponding mechanisms. Its asymmetric verification protocol keeps the common-path acceptance upload compact and moves richer, rejection-only correction information to the downlink. A total-variation (TV) certificate for the residual distribution determines whether a small target top-K response suffices; if not, the protocol progressively escalates through proposal-based exact recovery before falling back to the full distribution. Its confirmed-prefix pipeline exposes only independent, valid requests to the edge scheduler and lets the cloud re-batch arrived blocks, hiding verification waits when another confirmed-prefix request is ready without using same-request runahead. Across three draft-target pairs, two workloads, and three asymmetric network profiles, our end-to-end evaluation shows that AsymSpec delivers 2.82-28.03$\times$ the output-token throughput of the strongest baseline.
Innovation
Methods, ideas, or system contributions that make the work stand out.
speculative decoding
asymmetric networks
cloud-edge collaboration
total variation certificate
confirmed-prefix pipeline
🔎 Similar Papers
No similar papers found.
