ImpactHO: Importance-Aware KV Cache Transfer for Multi-User Edge LLM Handover

2026-08-11

AI Summary

This work addresses the challenge of maintaining inference continuity for edge-deployed large language models (LLMs) during multi-user handovers across edge nodes, where limited backhaul bandwidth prevents full migration of key-value (KV) caches. To mitigate this, the authors propose an importance-aware cache truncation mechanism that selectively transfers only high-value KV entries. They formulate multi-user cache allocation as a convex optimization problem aimed at maximizing average accuracy and, for the first time, extend the information-theoretic water-filling principle to the multi-user LLM handover setting. A closed-form weighted water-filling algorithm is developed for efficient online scheduling. Experimental results demonstrate that within a 500 ms transmission window, the system achieves an average accuracy of 93.7%, falling just 0.5 percentage points short of the full-cache upper bound and reaching 98.2%–99.5% of the clairvoyant optimum.

Abstract

Edge LLMs must preserve inference continuity when a user hands over between edge nodes, requiring key-value (KV) cache transfer to the target node. However, simultaneous handovers saturate the backhaul, preventing full cache delivery within the mobility-imposed transfer window. Rather than allocating bandwidth as if all cache entries were equally valuable, we order each user's KV cache by importance and transmit only its most informative fraction, turning token-level sparsity into communication savings. We cast the transfer as a multi-user backhaul allocation problem that maximizes average accuracy across users. Each user's partial-cache accuracy serves as its utility: a sigmoid that fits measurements on the RULER benchmark with $R^2>0.99$ across models and context lengths. Because importance ordering front-loads the high-value entries, the concave region of the accuracy curve spans nearly the entire cache. Our proposed allocator keeps served users within this region, making each per-slot allocation problem convex. The optimum is derived via a closed-form weighted water-filling solution that generalizes information-theoretic water-filling and enables online scheduling. The proposed allocator attains over 93.7% average accuracy in a 500ms transfer window, within 0.5pp of the full-cache ceiling, and reaches 98.2-99.5% of a clairvoyant upper bound.

Problem

• KV cache transfer
• multi-user handover
• edge LLM
• backhaul saturation
• inference continuity

Innovation

• importance-aware KV cache
• edge LLM handover
• multi-user backhaul allocation
• weighted water-filling
• token-level sparsity