NIXT: A NCCL Inspector Exporter Tool for Observability of Collective Communication in Large Model Training

2026-08-02

AI Summary

This work addresses the challenge of observing collective communication performance in large-scale model training, where limited visibility impedes efficient root-cause analysis of performance fluctuations. The authors present the first approach to transform NCCL Inspector-collected communication data into actionable insights by developing a lightweight export and analysis toolkit. This enables efficient observability and dynamic tracing of communication behavior across thousand-GPU clusters, accurately identifying straggler nodes. Integrating NCCL plugins, distributed monitoring, and performance data analytics, the method was successfully deployed during Nemotron-4 pretraining on a 2,048 H100 GPU cluster, effectively revealing the impact of parallelization strategies and scale on communication performance and substantially improving anomaly attribution efficiency.

Abstract

As machine learning workloads scale, it is increasingly important to gain more observability into the performance of collective communication to easily identify performance vari- ations and accelerate root cause identification. Towards this goal, the Nvidia Collective Communication Library (NCCL) introduced NCCL Inspector, a profiler plugin that provides lightweight and continuous reporting of NCCL communication performance statistics. However, the large volume of data collected by NCCL Inspector can be difficult to assess and to extract actionable insights from. This paper presents NIXT, a NCCL Inspector Exporter Tool that improves the observability of collective communication by providing readily accessible analysis and actionable insights from NCCL Inspector profiling. To highlight the benefits of our Exporter Tool, we present a case study of Nemotron-4 LLM pretraining on an Nvidia H100 GPU cluster with up to 2,048 GPUs, demonstrate observability into how communication phases change with ML parallelism and GPU scale, and perform attribution of performance variation and root cause analysis of stragglers.

Problem

• observability
• collective communication
• NCCL
• large model training
• performance analysis

Innovation

• NCCL Inspector
• collective communication
• observability
• large model training
• performance attribution