Generalization behavior of OPTQ and the role of regularization

2026-09-25

AI Summary

This study addresses the challenges of bounding generalization error and elucidating regularization mechanisms in neural network quantization. We systematically analyze the generalization behavior of OPTQ and its variants under test distributions, deriving theoretical bounds on the expected squared error. Through rigorous theoretical proofs and stochastic optimization analysis, we uncover how regularization terms influence dependence on the calibration set, and accordingly propose a novel regularization parameter selection strategy. Experimental results demonstrate that this strategy significantly outperforms existing methods, effectively reducing model quantization error.

Abstract

Large neural networks can be compressed by rounding or "quantizing" their weights to numbers that admit representations with fewer bits. One algorithm for quantization, OPTQ, progressively quantizes the weights of a neural network so that the squared quantization error on a specified calibration dataset is as small as possible. We study the performance of OPTQ and a variant algorithm, stochastic OPTQ, in a generalization setting and derive bounds for the expected squared error accrued by the algorithm when a test point is drawn from a fixed distribution. We prove two results. One result relates the generalization error to the error on a calibration dataset comprising independent samples from the same distribution as the test distribution. The other result bounds the generalization error of stochastic OPTQ for all sufficiently nice distributions, regardless of the calibration dataset. In both of these results, the regularization term $λ$ plays an important role. We use insights from these results to make a new recommendation for the choice of $λ$ and see that this choice of $λ$ preforms favorably in experiments when compared to prior recommendations in the literature.

Problem

• quantization
• generalization error
• OPTQ
• regularization
• neural network compression

Innovation

• OPTQ
• quantization
• generalization error
• regularization
• stochastic OPTQ