NUNA: Characterizing and Mitigating Non-Uniform Network Access in Multi-Die GPU Scale-Up Systems

2026-08-01

AI Summary

This study addresses the performance bottleneck imposed by non-uniform network access (NUNA)—a latency disparity arising from physical layout in multi-chip GPU systems—that critically hinders latency-sensitive inter-GPU communication. The work presents the first systematic characterization of NUNA and introduces a hardware-software co-designed approach comprising NUNA-aware routing (NAR) and placement (NAP). NAR optimizes communication paths through topology-aware routing, while NAP colocates thread blocks and data near I/O interfaces to minimize access latency. Experimental results demonstrate that NAP alone accelerates collective communication by up to 1.5×, and when combined with NAR, achieves a 1.8× speedup. In large model inference, the proposed techniques reduce per-token generation time by 7% on average, with improvements reaching 28% in the best case, thereby filling a critical gap in spatial communication optimization for multi-GPU systems.

Abstract

Graphics processing unit (GPU) architectures are growing in size to meet the increasing compute and memory requirements. As GPU sizes increase, intra-socket wire transfer delay increases significantly. While previous research has optimized for compute and memory locality within a socket, the spatial impact on inter-GPU communication has not been well-studied. We introduce the term non-uniform network access (NUNA) to describe this emerging optimization dimension in multi-GPU systems. We specifically focus on latency-sensitive collective communication, common in machine learning inference. First, we highlight the need for NUNA-aware routing (NAR), choosing optimized, spatially-aware inter-GPU paths in large scale-up network topologies. Second, we introduce NUNA-aware placement (NAP), placing threadblocks and data near I/O to optimize the inter-GPU traffic. We demonstrate that the NAP optimizations alone offer up to 1.5x collective speedups over a locality-unaware baseline. Combining NAP with NAR yields up to 1.8x faster collectives over the locality-unaware baseline. This leads to 7% mean (28% max) time per output token speedup in machine learning inference.

Problem

• Non-Uniform Network Access
• Multi-GPU Systems
• Collective Communication
• Latency Sensitivity
• GPU Scale-Up

Innovation

• NUNA
• NAR
• NAP
• multi-die GPU
• collective communication