HCCL: Collective Communication for Meta Training and Inference Accelerators

2026-07-31

AI Summary

This work addresses the performance bottleneck in large-scale AI training and inference caused by inefficient overlap between computation and communication, as well as high communication overhead. To this end, the authors develop a customized collective communication library for the Meta MTIA 300 accelerator, integrating a backend network within the chip package for the first time. By combining near-memory computing (NMC) with a dedicated message engine (ME), the design enables full communication offload. The paper introduces a compiler-driven communication model, topology-aware algorithms, and one-sided communication primitives tailored for inference, optimizing collective operations across heterogeneous scale-up and scale-out networks. Experiments demonstrate that, in training scenarios, intra-rack collective bandwidth reaches 940 GB/s with less than 0.5% impact on concurrent compute throughput; in inference, communication latency is significantly reduced, greatly enhancing compute-communication pipeline efficiency.

Abstract

We present HCCL, a collective communication library co-designed with Meta's MTIA 300 accelerator, the first Meta chip to integrate backend networking directly on chip package. MTIA 300 includes dedicated message engines (MEs) with near-memory compute (NMC) that fully offload collective execution from the compute grid, enabling large overlap between computation and communication. HCCL uses a compiled communication model in which the host generates a complete description of each collective including dependencies. We describe the control and data path architecture, topology-aware algorithm selection across MTIA 300's asymmetric scale-up and scale-out network, and optimizations for both training and inference workloads. For training, HCCL achieves up to 940 GB/s on intra-rack collectives while introducing less than 0.5% degradation to concurrent compute throughput. For inference, we leverage one-sided communication primitives that bypass the scheduling path to minimize collective latency and describe collective designs that improve compute-communication pipelining for latency-sensitive workloads.

Problem

• collective communication
• accelerator
• training
• inference
• compute-communication overlap

Innovation

• collective communication
• near-memory compute
• compiled communication model
• topology-aware algorithm
• one-sided communication