A Shape-Adaptive Architecture with Disaggregated Quantization for Efficient LLM Serving

2026-10-05

AI Summary

This study addresses the inefficiency of existing accelerators in handling dynamic workloads across the prefill and decode phases of LLM inference. To this end, we propose DynaCore, a system-architecture co-design featuring the first shape-adaptive compute units that optimize spatiotemporal dimensions via systolic array reconfiguration and split-K reduction mapping. Additionally, it introduces a dual-end/weight-only decoupled quantization scheme with an equal-width mixed-precision datapath, alongside a runtime scheduling framework for efficient adaptation. Evaluations on real-world serving traces demonstrate that DynaCore reduces time-to-first-token (TTFT) by 3.5× and time-per-output-token (TPOT) by 36.55×, significantly outperforming existing solutions.

Abstract

Large language models (LLMs) have become the backbone of modern AI applications, but pose significant challenges for efficient inference. Their autoregressive generation divides execution into two phases: prefill, dominated by large GEMMs, and decoding, dominated by small GEMVs. Modern serving systems further introduce complexity through continuous batching and prefill-decoding disaggregation, leading to dynamic workloads and phase separation. However, existing accelerators remain poorly aligned with these system-level behaviors, resulting in inefficiencies in LLM serving. In this work, we present DynaCore, a unified architecture for efficient LLM serving via system-architecture co-design. We observe that the compute tile a systolic array executes, its Minimum Efficient Unit (MEU), spans all three GEMM dimensions. DynaCore reshapes the MEU along all three: spatially it trades array width against height asymmetrically, raising weight delivery while leaving the input path untouched, and temporally Split-K maps the reduction onto the array, folding partial sums through the interconnect the array already has. To exploit phase separation, we further propose disaggregated quantization, applying dual-side quantization to prefill and weight-only quantization to decoding, with an inner-product mixed-precision datapath that keeps output width invariant to precision. A runtime scheduling framework then selects an MEU per batch. Evaluation with real-world serving traces shows that DynaCore substantially reduces service-level latency over quantization and reconfigurable accelerators, improving TTFT by 3.50x and 2.97x and TPOT by 36.55x and 8.02x, respectively.

Problem

• Large Language Models
• LLM Serving
• Hardware Accelerator
• Inference Efficiency
• Prefill-Decoding Disaggregation

Innovation

• Shape-Adaptive Architecture
• Disaggregated Quantization
• Minimum Efficient Unit (MEU)
• System-Architecture Co-design
• LLM Serving