3DLS: A 3D Logic-Stacked Architecture for Disaggregated LLM Serving

2026-07-01

AI Summary

This work addresses the performance degradation in traditional 2D/2.5D chiplet architectures for large language model (LLM) inference, where KV cache transfers during the prefill phase and tensor-parallel communications during decoding contend for the same lateral interconnect, causing increased latency and reduced throughput. To resolve this, the authors propose 3DLS, a novel architecture that introduces physical traffic isolation in a prefill-decode decoupled LLM serving system. Leveraging a logically stacked 3D chiplet design, 3DLS dedicates vertical interconnects exclusively to KV cache transmission while reserving lateral interconnects for tensor-parallel communication, thereby achieving strict physical separation of these two critical traffic types. Experimental results demonstrate that 3DLS improves throughput by 1.49× and reduces end-to-end latency by 60.2% compared to a shared-interconnect baseline, and outperforms a priority-based management scheme by 1.17× in throughput and 31.4% in latency reduction.

Abstract

Large language model (LLM) serving increasingly combines prefill-decode (PD) disaggregation with tensor parallelism (TP) to support large models and long contexts. In conventional 2D/2.5D chiplet architectures, layer-wise prefill-to-decode KV-cache transfer decode-side TP collectives share the same lateral die-to-die (D2D) interconnect, creating mixed-traffic contention on the decode critical path. This contention increases communication latency, prolongs token generation intervals, and degrades end-to-end serving performance. We propose 3DLS, a logic-on-logic 3D-stacked chiplet architecture that separates traffic classes by routing KV-cache transfers through vertical interconnects while preserving decode-side TP collectives on the lateral D2D fabric. 3DLS achieves up to 1.49$\times$ throughput and 60.2\% lower end-to-end (E2E) latency over the shared-fabric planar baseline, and still achieves up to 1.17$\times$ throughput and 31.4\% lower E2E latency over a workload-aware priority-managed planar baseline. These results highlight that physical isolation is an important design principle for future chiplet-based PD-disaggregated LLM serving systems.

Problem

• LLM serving
• prefill-decode disaggregation
• KV-cache transfer
• tensor parallelism
• chiplet architecture

Innovation

• 3D-stacked architecture
• KV-cache transfer
• prefill-decode disaggregation
• tensor parallelism
• chiplet interconnect