Smooth Approximations of the Rounding Function

2025-04-26

AI Summary

This paper addresses the fundamental challenge that classical rounding functions are non-differentiable, hindering their integration into gradient-based optimization and machine learning frameworks. To resolve this, we propose two analytically differentiable, smooth approximations to rounding: (1) a local sigmoid-window function centered at integers, and (2) a weighted density-integration model based on the sigmoid derivative. Both approximations converge pointwise to the standard rounding function, achieving high approximation accuracy while maintaining O(1) computational complexity. To our knowledge, this work presents the first rigorously differentiable, arbitrarily precise rounding surrogate—overcoming the long-standing differentiability barrier of discrete rounding operations. Extensive experiments demonstrate substantial improvements in training stability and model performance across neural network quantization learning and discrete optimization tasks.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Mixed Discrete/Continuous Search
• Machine Learning: Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Responsible Web: Measurement, analysis, and circumvention of Web censorship

Abstract

We propose novel smooth approximations to the classical rounding function, suitable for differentiable optimization and machine learning applications. Our constructions are based on two approaches: (1) localized sigmoid window functions centered at each integer, and (2) normalized weighted sums of sigmoid derivatives representing local densities. The first method approximates the step-like behavior of rounding through differences of shifted sigmoids, while the second method achieves smooth interpolation between integers via density-based weighting. Both methods converge pointwise to the classical rounding function as the sharpness parameter k tends to infinity, and allow controlled trade-offs between smoothness and approximation accuracy. We demonstrate that by restricting the summation to a small set of nearest integers, the computational cost remains low without sacrificing precision. These constructions provide fully differentiable alternatives to hard rounding, which are valuable in contexts where gradient-based methods are essential.

Problem

• Proposes smooth approximations for rounding function in optimization
• Enables differentiable alternatives for gradient-based machine learning
• Balances smoothness and accuracy with controllable trade-offs

Innovation

• Smooth sigmoid windows approximate rounding steps
• Density-weighted sigmoid derivatives enable interpolation
• Controlled trade-off between smoothness and accuracy