Configurable and Hierarchical Allreduce

2026-08-03

AI Summary

This work addresses the performance bottlenecks of MPI_Allreduce under small-to-medium message sizes in large-scale systems, where latency, synchronization depth, and hardware hierarchy impose significant constraints. To overcome these limitations, the authors propose CHIARA, a configurable hierarchical Allreduce algorithm that innovatively integrates a rotating-root channel mechanism with a Rabenseifner-style hybrid approach. CHIARA models hardware hierarchy through a logical batch-channel topology, employs phased scheduling to activate only subsets of reduction vectors, and maintains channel-aligned intermediate layouts at the Reduce-Scatter/Allgather boundary to eliminate centralized bottlenecks and avoid redundant intra-domain reorganization. Experimental results demonstrate that CHIARA achieves speedups of up to 1.94×, 13.43×, and 13.48× on Polaris, Aurora, and Fugaku systems, respectively, and accelerates an end-to-end k-means application by up to 2.2×.

Abstract

MPI_Allreduce is among the most performance-critical collectives in large-scale scientific computing and distributed machine learning, yet the small- and medium-message regime remains challenging: latency, synchronization depth, and strong hardware hierarchy between intra- and inter-domain communication all compound per-invocation cost. We present CHIARA, a configurable hierarchical Allreduce that encodes hardware hierarchy through a logical batch-lane topology and executes a staged schedule in which only a bounded portion of the reduction vector is active at a time. Inter-batch communication is distributed across multiple ranks via a rotating-root lane primitive, avoiding centralized leaders. Tool further enables a semi-composed Rabenseifner-style Allreduce by preserving a lane-aligned intermediate layout across the Reduce-Scatter/Allgather boundary, eliminating redundant intra-domain reorganization. We evaluate Tool on Polaris, Aurora, and Fugaku, achieving speedups of up to 1.94x, 13.43x, and 13.48x over vendor MPI_Allreduce, and up to 2.2x end-to-end speedup in a parallel k-means application.

Problem

• MPI_Allreduce
• small-message communication
• hardware hierarchy
• latency
• synchronization depth

Innovation

• hierarchical Allreduce
• configurable collective
• batch-lane topology
• rotating-root lane
• semi-composed Rabenseifner