🤖 AI Summary
This work addresses the challenges of high-throughput, memory-constrained inference on general-purpose NPUs, where scheduling order, memory layout, and compute-DMA pipelining are tightly coupled, often causing excessive on-chip memory occupancy, redundant off-chip accesses, and suboptimal pipeline utilization under conventional phased compilation. To overcome these limitations, we propose DAN-Scheduler, the first deterministic offline co-optimization framework that jointly optimizes execution scheduling, memory placement, and pipeline overlapping through three integrated stages: Memory-Pressure-Aware Topological Sorting (MPAS), Deterministic Linear Repacking (DLR), and Critical Path Enhancement (CPE). Evaluated across six real-world Davinci NPU workloads, DAN-Scheduler reduces peak memory usage by 18.3%, DDR traffic by 20.4%, spill counts by 14.2%, and makespan by 16.3% on average compared to the best baseline, substantially improving resource efficiency.
📝 Abstract
Neural Processing Units (NPUs) are increasingly deployed for high-throughput, memory-constrained inference, yet their hierarchical on-chip memories and heterogeneous compute and data-movement engines tightly couple execution order, memory placement, and pipeline overlap. Existing compiler flows often optimize these dimensions separately, causing excessive on-chip residency, unnecessary off-chip traffic, and underutilized pipelines.We present DAN-Scheduler, a deterministic offline scheduling and compiler optimization framework for intra-core NPU execution. It co-optimizes these decisions in three stages. Memory-Pressure-Aware Topological Scheduling (MPAS) reorders operators to shorten tensor lifetimes and reduce peak on-chip memory usage. Deterministic Linear Repackaging (DLR) builds conflict-free memory layouts and applies a tier-aware, cost-aware spill heuristic to reduce fragmentation and off-chip traffic under limited capacity. Critical Path Enhancement (CPE) improves compute-DMA overlap while preserving the memory behavior established by the first two stages.
We evaluate DAN-Scheduler on six trace-derived operator-level DAGs collected from a real Davinci NPU and replayed on a generalized NPU execution model. Against four strong external baselines, DAN-Scheduler achieves the best or tied-best result on all 24 workload-metric cells, reducing peak memory, extra DDR traffic, spill count, and makespan by 18.3%, 20.4%, 14.2%, and 16.3% on average over the best external competitor. Relative to the original schedule, it reduces the same metrics by 38.3%, 62.0%, 64.9%, and 57.5%. These results show that deterministic stage-wise co-optimization is effective for memory-constrained NPU execution. Code and data are available at https://anonymous.4open.science/r/MICRO2026-5C74