Mnemosyne: Parallelization Strategies for Efficiently Serving Multi-Million Context Length LLM Inference Requests Without Approximations

2024-09-25

arXiv.org

AI Summary

Large language models (LLMs) face significant challenges—including high time-to-first-token (TTFT) and time-between-tokens (TBT), head-of-line blocking, and low hardware utilization—when performing interactive inference over ultra-long contexts (up to 10M tokens). Method: We propose the first lossless long-context inference system, built upon a novel 3D parallel architecture integrating adaptive chunking, sequence-level pipeline parallelism (SPP), and KV-cache parallelism (KVP), coupled with hybrid request batching and memory-compute co-scheduling. Contribution/Results: Our system achieves production-grade performance on 10M-token contexts: TBT ≤ 30 ms, substantially reduced TTFT, and improved throughput—all without approximation, compression, or accuracy loss. To our knowledge, it is the only inference service system supporting multi-million-token contexts while simultaneously ensuring zero precision degradation, high throughput, and low latency.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

As large language models (LLMs) evolve to handle increasingly longer contexts, serving inference requests for context lengths in the range of millions of tokens presents unique challenges. While existing techniques are effective for training, they fail to address the unique challenges of inference, such as varying prefill and decode phases and their associated latency constraints - like Time to First Token (TTFT) and Time Between Tokens (TBT). Furthermore, there are no long context inference solutions that allow batching requests to increase the hardware utilization today. In this paper, we propose three key innovations for efficient interactive long context LLM inference, without resorting to any approximation: adaptive chunking to reduce prefill overheads in mixed batching, Sequence Pipeline Parallelism (SPP) to lower TTFT, and KV Cache Parallelism (KVP) to minimize TBT. These contributions are combined into a 3D parallelism strategy, enabling Mnemosyne to scale interactive inference to context lengths at least up to 10 million tokens with high throughput enabled with batching. To our knowledge, Mnemosyne is the first to be able to achieve support for 10 million long context inference efficiently, while satisfying production-grade SLOs on TBT (30ms) on contexts up to and including 10 million.

Problem

• Efficiently serving multi-million token context LLM inference requests
• Addressing varying prefill and decode phase latency constraints
• Eliminating head-of-line blocking in long-context inference systems

Innovation

• Adaptive chunking with slack-aware scheduling
• Sequence Pipeline Parallelism (SPP)
• KV Cache Parallelism (KVP)