Local Stability and Gaussian Smoothing of Quantized Neural Networks

2026-07-22

AI Summary

This work addresses the instability in training and inference of quantized neural networks caused by discontinuous activation functions. We propose Gaussian smoothing as a smooth surrogate and establish dimension-dependent error bounds between the original and smoothed functions under locally bounded oscillation conditions. For the first time, we integrate Gaussian smoothing with the local stability theory of quantized networks. Closed-form expressions for the Gaussian averages of ReLU and sign functions are derived, enabling a layer-wise smoothing mechanism based on pre-activation aggregation. By explicitly modeling quantization noise, our approach ensures consistency between smooth gradients during training and discrete operations at inference. Experiments on high-dimensional binary perceptrons validate the efficacy of the method, offering both theoretical grounding and a practical gradient approximation scheme for quantized neural networks.

Abstract

We study Gaussian averaging as a smooth surrogate for quantized neural models. Under bounded local oscillation, we derive a local dimension-dependent bound on |f-g|, linking Gaussian smoothing to the stability analysis of discontinuous networks. We compute closed-form Gaussian averages of the rectified linear unit (ReLU) and sign activation functions, and illustrate the mechanism on a high-dimensional binary perceptron, where layer-preactivation aggregation under an explicit quantization-noise surrogate yields the Gaussian envelope used in inference-side smoothing and training-side smooth surrogate gradients.

Problem

• Quantized Neural Networks
• Local Stability
• Gaussian Smoothing
• Discontinuous Networks
• Bounded Local Oscillation

Innovation

• Gaussian smoothing
• quantized neural networks
• local stability
• smooth surrogate
• closed-form averaging