Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Management

2025-01-12

AI Summary

To address the high KV cache memory overhead and GPU load imbalance in large-scale LLM serving, this paper proposes a multi-GPU collaborative scheduling framework. Methodologically, it introduces (1) the first online joint scheduling algorithm for multiple LLMs across multiple GPUs that supports request migration; (2) an adaptive request migration mechanism guided by real-time GPU load and KV cache state awareness, achieving cross-GPU load balancing under strict constraints on migration frequency; and (3) coordinated KV cache management and optimized high-speed inter-GPU communication. Experimental results demonstrate that, compared to state-of-the-art systems, the framework reduces GPU resource consumption by 31%, improves peak GPU utilization by up to 43%, and significantly enhances system scalability and resource efficiency.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Serving large language models (LLMs) for massive users is challenged by the significant memory footprint of the transient state, known as the key-value (KV) cache, which scales with sequence length and number of requests. Instead of renting or buying more expensive GPUs, the load imbalance of the KV cache across GPUs, coupled with recent advances in inter-GPU communication, provides an opportunity to serve more requests via request migration. However, high migration overhead and unpredictable request patterns make it challenging. Therefore, this paper proposes MELL, a memory-efficient LLM serving system via multi-GPU KV cache management. It saves the number of GPUs needed in the system by considering the dynamic KV cache load and the costly request migration. Specifically, we first develop an adaptive request migration mechanism to balance the computational and communication overheads and adapt to diverse resource conditions. Then, we design an online algorithm tailored to a multi-LLM request and multi-GPU scheduling problem with migration enabled. It aims to minimise the required GPUs while limiting the number of migrations. Finally, we implement a prototype of MELL and demonstrate that it reduces the number of GPUs by 31% and increases the GPU utilization by 43% at most compared to existing LLM serving systems.

Problem

• Large Language Models
• Memory Management
• Efficiency

Innovation

• MELL System
• Adaptive Migration Mechanism
• Dedicated Scheduling Algorithm