V-Rex: Real-Time Streaming Video LLM Acceleration via Dynamic KV Cache Retrieval

2025-12-13

AI Summary

Streaming video large language models (LVLMs) face critical challenges in edge-device real-time inference: KV cache size grows linearly with frame count, causing memory explosion, computational redundancy, and accuracy degradation—exacerbated by iterative prefilling. This paper proposes ReSV, the first training-agnostic dynamic KV cache retrieval algorithm, integrated with a hardware–software co-designed acceleration architecture. ReSV comprises spatiotemporal similarity–driven token clustering, bit-level compute units, an early-exit mechanism, and a customized Dynamic Retrieval Engine (DRE) for KV caching. Evaluated on edge devices, ReSV achieves 3.9–8.3 FPS real-time inference, improves energy efficiency by 3.1–18.5×, and boosts throughput by 1.9–19.7×, with negligible accuracy loss. The DRE occupies only 2.2% of total power consumption and 2.0% of chip area.

Technology Category

• Computer Vision: Image and Video Retrieval
• Machine Learning: Hardware-aware ML
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Systems and Infrastructure for Web, Mobile and WoT: Cloud, edge and content delivery systems for the Web

Abstract

Streaming video large language models (LLMs) are increasingly used for real-time multimodal tasks such as video captioning, question answering, conversational agents, and augmented reality. However, these models face fundamental memory and computational challenges because their key-value (KV) caches grow substantially with continuous streaming video input. This process requires an iterative prefill stage, which is a unique feature of streaming video LLMs. Due to its iterative prefill stage, it suffers from significant limitations, including extensive computation, substantial data transfer, and degradation in accuracy. Crucially, this issue is exacerbated for edge deployment, which is the primary target for these models. In this work, we propose V-Rex, the first software-hardware co-designed accelerator that comprehensively addresses both algorithmic and hardware bottlenecks in streaming video LLM inference. At its core, V-Rex introduces ReSV, a training-free dynamic KV cache retrieval algorithm. ReSV exploits temporal and spatial similarity-based token clustering to reduce excessive KV cache memory across video frames. To fully realize these algorithmic benefits, V-Rex offers a compact, low-latency hardware accelerator with a dynamic KV cache retrieval engine (DRE), featuring bit-level and early-exit based computing units. V-Rex achieves unprecedented real-time of 3.9-8.3 FPS and energy-efficient streaming video LLM inference on edge deployment with negligible accuracy loss. While DRE only accounts for 2.2% power and 2.0% area, the system delivers 1.9-19.7x speedup and 3.1-18.5x energy efficiency improvements over AGX Orin GPU. This work is the first to comprehensively tackle KV cache retrieval across algorithms and hardware, enabling real-time streaming video LLM inference on resource-constrained edge devices.

Problem

• Reduces KV cache memory growth in streaming video LLMs
• Addresses computational and data transfer bottlenecks in real-time inference
• Enables efficient edge deployment with minimal accuracy loss

Innovation

• Dynamic KV cache retrieval algorithm reduces memory usage
• Hardware accelerator with bit-level and early-exit computing units
• Software-hardware co-design enables real-time edge video LLM inference