Every Microsecond Matters: Achieving Near Speed-of-Light Latency in GPU Collectives

2026-07-17

AI Summary

This work addresses the microsecond-scale latency bottlenecks in delay-sensitive applications—such as long-context large language model (LLM) inference—where existing GPU collective communication schemes over-optimize for bandwidth while neglecting latency. The authors propose low-latency, custom collective communication kernels built upon NCCL’s device-side API, introducing symmetric collective primitives within NCCL for the first time. Key innovations include barrier-free synchronization, symmetric memory access patterns, and efficient multicast utilization. Microbenchmark evaluations demonstrate that communication latency for small-to-medium messages is reduced to within 7% of the hardware’s speed-of-light lower bound. When applied to LLM inference, the approach significantly lowers per-token latency and improves throughput, while also accelerating HPC workloads such as cuSOLVERMp, closely approaching theoretical latency limits.

Abstract

GPU collective communication is typically optimized for bandwidth, yet many emerging workloads are increasingly limited by latency. Long-context decode-heavy large language model (LLM) inference is a prime example, where serving large models requires multiple GPUs, and many small collectives lie directly on the critical path of token generation. Therefore, even microsecond of overhead can impact performance and cost. In this work, we study how to approach the hardware Speed-of-Light (SoL) lower bound for GPU collectives within a scale-up network. We identify key principles for near-optimal designs, including barrier-free synchronization and efficient use of symmetric memory and multicast. Building on NCCL's device-side API, we develop low-latency interfaces for constructing custom collective kernels and use them to implement new symmetric collectives in NCCL. Microbenchmarks show substantial latency reductions for small and medium messages, reducing overhead to within 7% of the absolute SoL lower bound. When integrated into real applications, these kernels improve inter-token latency and throughput in LLM inference and accelerate cuSOLVERMp, demonstrating benefits for both AI inference and traditional HPC workloads.

Problem

• GPU collective communication
• latency optimization
• large language model inference
• Speed-of-Light lower bound
• inter-token latency

Innovation

• near-speed-of-light latency
• barrier-free synchronization
• symmetric collectives
• NCCL device-side API
• GPU collective communication