Beyond Prefill-Decode Disaggregation: Dissecting LLM Inference for Heterogeneous Platforms via Dynamic Operator Scheduling

2026-07-28

AI Summary

This work addresses the challenge of efficiently optimizing end-to-end latency for large language model inference on heterogeneous platforms—such as NPUs and processing-in-memory (PIM) systems—where existing prefill-decode separation and Roofline-based operator placement strategies fall short due to dynamic workloads, device contention, and suboptimal weight layouts. To overcome these limitations, the authors propose DOPS, a framework that jointly optimizes dynamic operator scheduling and tiled weight layout through a stage-aware directed acyclic graph. Its key innovations include the Bifocal scheduler, which enables closed-loop dynamic operator assignment, and the Weight Layout Arbiter (WLA), which selects hardware-efficient weight layouts under strict memory constraints. Experiments demonstrate that Bifocal achieves 1.20–2.23× speedup over baselines, with WLA providing an additional 1.28–1.33× acceleration, substantially reducing inference latency while enabling hardware scalability analysis.

Abstract

Prefill-decode disaggregation (PD) and roofline-based operator placement are common strategies for partitioning Large Language Model (LLM) inference across heterogeneous systems, but they are often insufficient in practice. End-to-end latency also depends on workload shape, runtime device contention, and persistent weight layout. We present DOPS (dynamic operator scheduling), a hardware-aware, closed-loop framework that jointly optimizes operator scheduling and blockwise weight layouts. DOPS constructs a stage-aware directed acyclic graph (DAG) and integrates two components: the Bifocal scheduler for dynamic operator-to-device placement and the Weight Layout Arbiter (WLA) for selecting hardware-efficient weight layouts under strict memory constraints. Across representative heterogeneous systems combining neural processing units (NPUs) and processing-in-memory (PIM) devices, Bifocal achieves geometric-mean speedups of 1.20$\times$ to 2.23$\times$ over the PD baseline. WLA provides an additional geometric-mean speedup of 1.28$\times$ to 1.33$\times$ over Bifocal/Linear. DOPS also supports systematic analysis of workload sensitivity and hardware scalability for LLM serving. The source code is available at https://github.com/YIAI-02/TriForm, and the visualization tool is demonstrated at https://youtu.be/Ya_oMCyYno0.

Problem

• LLM inference
• heterogeneous platforms
• operator scheduling
• weight layout
• end-to-end latency

Innovation

• Dynamic Operator Scheduling
• Heterogeneous Platforms
• Weight Layout Optimization
• LLM Inference
• Hardware-aware Scheduling