Optimizing SLO-oriented LLM Serving with PD-Multiplexing

2025-04-20

AI Summary

Addressing the dual challenges of stringent SLO guarantees and high throughput in LLM serving—specifically across the prefill and decode phases—this work identifies a fundamental trade-off: geo-distributed partitioning meets SLOs but forfeits KV cache reuse, while in-place sharing improves throughput yet suffers from low resource utilization and high scheduling overhead due to tight phase coupling. To resolve this, we propose Phase-Decoupled (PD), a multi-tenant multiplexing architecture that enables spatially adaptive, per-phase resource allocation and in-situ KV cache sharing on a single GPU. PD incorporates adaptive group scheduling, contention-free performance modeling, and SLO-aware request dispatching. Experiments demonstrate an average 5.1× throughput improvement over baselines, with peak gains up to 17.5×, while achieving 100% SLO compliance under complex, multi-turn workloads.

Technology Category

• Search and Optimization: Distributed Search
• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Hardware-aware ML

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Modern LLM services demand high throughput and stringent SLO guarantees across two distinct inference phases-prefill and decode-and complex multi-turn workflows. However, current systems face a fundamental tradeoff: out-of-place compute partition enables per-phase SLO attainment, while in-place memory sharing maximizes throughput via KV cache reuse. Moreover, existing in-place compute partition also encounters low utilization and high overhead due to phase-coupling design. We present Yoda, a new LLM serving framework that resolves this tension via PD multiplexing, enabling in-place and phase-decoupled compute partition. Yoda leverages low-level GPU partitioning techniques to multiplex prefill and decode phases spatially and adaptively on shared GPUs, while preserving in-place memory sharing. To fully leverage the multiplexing capability, Yoda introduces an adaptive gang scheduling mechanism, a contention-free modeling method, and a SLO-aware dispatching policy. Evaluation shows that Yoda achieves an average $5.1 imes$ throughput improvement (up to $17.5 imes$) over state-of-the-art baselines, while consistently meeting SLO targets under complex LLM workloads.

Problem

• Resolving throughput-SLO tradeoff in LLM serving
• Overcoming low utilization in phase-coupled compute partition
• Enabling efficient KV cache reuse with multiplexing

Innovation

• PD multiplexing enables in-place compute partition
• Low-level GPU partitioning for spatial multiplexing
• Adaptive gang scheduling and SLO-aware dispatching