Learning to optimize with guarantees: a complete characterization of linearly convergent algorithms

2025-08-01

AI Summary

Linearly convergent optimization algorithms often exhibit suboptimal average-case performance on practical problem instances, yet their worst-case convergence guarantees—critical for safety-critical applications—must be preserved. Method: We propose a safety-enhancing framework grounded in nonsmooth composite optimization, which, for the first time, fully characterizes the class of admissible update-rule transformations that preserve linear convergence. This unified characterization encompasses foundational algorithms—including gradient descent, Nesterov’s accelerated method, and projection-based methods—as special cases. Contribution/Results: Our framework strictly maintains theoretical linear convergence rates while significantly accelerating empirical convergence in demanding settings such as ill-conditioned linear systems solving and model predictive control. Experiments demonstrate substantial improvements in average solution efficiency under finite iteration budgets, achieving both rigorous worst-case guarantees and strong practical performance.

Technology Category

• Search and Optimization: Non-convex Optimization
• Machine Learning: Optimization
• Natural Language Processing: Safety and Robustness

Application Category

• Security and Privacy: Large-scale security measurements
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

In high-stakes engineering applications, optimization algorithms must come with provable worst-case guarantees over a mathematically defined class of problems. Designing for the worst case, however, inevitably sacrifices performance on the specific problem instances that often occur in practice. We address the problem of augmenting a given linearly convergent algorithm to improve its average-case performance on a restricted set of target problems - for example, tailoring an off-the-shelf solver for model predictive control (MPC) for an application to a specific dynamical system - while preserving its worst-case guarantees across the entire problem class. Toward this goal, we characterize the class of algorithms that achieve linear convergence for classes of nonsmooth composite optimization problems. In particular, starting from a baseline linearly convergent algorithm, we derive all - and only - the modifications to its update rule that maintain its convergence properties. Our results apply to augmenting legacy algorithms such as gradient descent for nonconvex, gradient-dominated functions; Nesterov's accelerated method for strongly convex functions; and projected methods for optimization over polyhedral feasibility sets. We showcase effectiveness of the approach on solving optimization problems with tight iteration budgets in application to ill-conditioned systems of linear equations and MPC for linear systems.

Problem

• Enhancing average-case performance while preserving worst-case guarantees
• Characterizing linearly convergent algorithms for nonsmooth composite optimization
• Tailoring optimization algorithms for specific applications like MPC

Innovation

• Augmenting algorithms for better average-case performance
• Preserving worst-case guarantees in optimization
• Characterizing linearly convergent algorithm modifications