Stochastic Rounding Increases Small Singular Values

2026-05-29

AI Summary

This study investigates whether stochastic rounding (SR) retains its regularizing effect in matrices with constant aspect ratios and examines its impact on the singular value spectrum. By integrating singular value analysis, a stochastic rounding quantization model, and spectral theory, the work demonstrates for the first time that SR not only enhances the smallest singular value but also collectively elevates multiple singular values in the tail of the spectrum. This finding reveals that the regularizing influence of SR extends beyond extreme aspect ratio regimes. The results establish SR as a universal spectral regularization mechanism, thereby broadening its theoretical foundation and application potential in numerical computation and low-precision machine learning.

Abstract

Over the past half-dozen years, stochastic rounding (SR) has regained significant attention as a quantization scheme for low-precision floating-point arithmetic, with applications spanning numerical analysis and modern machine learning systems. Recent work has shown that SR acts as an implicit regularizer by increasing the smallest singular value of extremely tall-and-thin (or, symmetrically, short-and-fat) matrices. In this work, we substantially sharpen and extend this understanding in two directions. First, we show that the regularization effect of SR is not restricted to extreme aspect ratio regimes: it persists for matrices with constant aspect ratio. Second, we demonstrate that SR does not merely regularize the smallest singular value, but instead lifts entire clusters of singular values at the tail of the spectrum. Together, these results provide a more general characterization of stochastic rounding as a spectral regularizer, revealing that its effects extend beyond extremal aspect ratios and act on a broader portion of the singular value spectrum.

Problem

• stochastic rounding
• singular values
• spectral regularization
• matrix aspect ratio
• low-precision arithmetic

Innovation

• stochastic rounding
• singular values
• spectral regularization
• matrix aspect ratio
• low-precision arithmetic