Technology Category
Machine Learning: OptimizationSearch and Optimization: Non-convex OptimizationReasoning under Uncertainty: Stochastic Optimization
Convergence of a class of gradient-free optimisation schemes when the objective function is noisy, irregular, or both
🤖 AI Summary
This paper investigates the convergence of gradient-free optimization algorithms under noisy, nonsmooth, or both settings—typical in black-box optimization where gradients are unavailable. We propose a unified analytical framework integrating model-based strategies and smoothing techniques, replacing gradient estimation for nonsmooth or stochastic objectives with smooth approximations via a generalized gradient descent recursion. Under minimal regularity assumptions—requiring only local bounded variation—we rigorously establish convergence guarantees for both deterministic and stochastic settings. Our analysis uncovers a fundamental trade-off between the smoothing parameter and step size, characterizing their joint impact on convergence rate and stability. Extensive experiments on diverse machine learning classification tasks demonstrate the method’s effectiveness and robustness against noise and nonsmoothness.
Technology Category
Application Category
📝 Abstract
We investigate the convergence properties of a class of iterative algorithms designed to minimize a potentially non-smooth and noisy objective function, which may be algebraically intractable and whose values may be obtained as the output of a black box. The algorithms considered can be cast under the umbrella of a generalised gradient descent recursion, where the gradient is that of a smooth approximation of the objective function. The framework we develop includes as special cases model-based and mollification methods, two classical approaches to zero-th order optimisation. The convergence results are obtained under very weak assumptions on the regularity of the objective function and involve a trade-off between the degree of smoothing and size of the steps taken in the parameter updates. As expected, additional assumptions are required in the stochastic case. We illustrate the relevance of these algorithms and our convergence results through a challenging classification example from machine learning.
Problem
Research questions and friction points this paper is trying to address.
Analyzing convergence of gradient-free methods for noisy or non-smooth functions
Studying iterative algorithms for black-box, intractable objective optimization
Exploring smoothing and step-size trade-offs in zero-th order optimization
Innovation
Methods, ideas, or system contributions that make the work stand out.
Gradient-free optimization for noisy, irregular functions
Generalized gradient descent with smoothed approximations
Model-based and mollification methods for zero-th order optimization
🔎 Similar Papers
No similar papers found.
