A Second-Moment Theory for Floating-Point Reduction Trees

2026-07-21

AI Summary

This work addresses the dependence of summation error in floating-point reduction trees on the order of partial sums—a dependency not captured by traditional worst-case bounds. Under a conditionally unbiased rounding assumption, the paper establishes the first precise second-moment error model for binary reduction trees, introducing a common-ancestor kernel \( K_T \) and structural statistics \( \Lambda_1, \Lambda_2 \) to quantitatively characterize how input distributions and tree topology jointly influence error. The theory reveals error scaling behaviors for balanced trees, sequential reductions, and blocked strategies, and extends naturally to matrix multiplication (GEMM). Validated through stochastic rounding models, graph-kernel analysis, and multi-precision experiments (binary64/32/16, bfloat16), the model accurately predicts error rankings across tree structures, achieves GEMM prediction errors below 3%, and successfully reproduces RMS error trends observed in real-world array libraries.

Abstract

Summation error depends on partial-sum order, which standard worst-case bounds omit. To capture this dependence, we derive an exact mean-square error (MSE) recurrence for a binary reduction tree T under conditionally unbiased rounding. With unit roundoff u, the constant-nu model sets the local variance at pre-rounding value x to nu u^2 x^2. Its leading tree-dependent cost for the input vector p is p^T K_T p, where the common-ancestor kernel K_T counts the internal ancestors shared by each pair of leaves. For i.i.d. inputs of mean mu and variance tau^2, this expected cost is tau^2 Lambda_1(T) + mu^2 Lambda_2(T), where Lambda_1 is total leaf depth and Lambda_2 sums squared internal-subtree sizes; Lambda_1 governs centered inputs, while Lambda_2 captures nonzero means. We use these statistics to characterize optimal tree topologies and schedules. Balanced and sequential trees attain the centered extrema. For k inputs, optimal two-stage sequential blocking yields root-mean-square (RMS) error scaling as k^{3/4}. For fixed-stage hierarchies, geometric schedules are optimal for centered inputs, whereas the optimal noncentered stage exponents halve successively. For independent centered inputs with unequal variances, Huffman coding minimizes variance-weighted depth over free leaf assignments. We extend the kernel to matrix multiplication through operand Gram matrices. We then test the approximation under round-to-nearest using exact residuals. Across binary64, binary32, and software-emulated binary16 and bfloat16, the model recovers the ordering among tree topologies; K_T tracks AR(1) partial-sum costs. For GEMM, independently calibrated predictions differ from measurements by at most 3% on the tested grid. A reduction tree extracted from an array library predicts the measured RMS scaling. However, stagnation and bias in positive low-precision sums limit the model's applicability.

Problem

• floating-point summation
• reduction trees
• rounding error
• mean-square error
• numerical stability

Innovation

• second-moment analysis
• floating-point summation
• reduction tree
• common-ancestor kernel
• rounding error modeling