DADA: Dual Averaging with Distance Adaptation

2025-01-17

AI Summary

We address the general convex optimization problem—requiring algorithms that neither presuppose the Lipschitz constant, the smoothness order of the objective, nor the total number of iterations, and that uniformly handle unconstrained problems, constrained problems (including unbounded domains), and broad function classes ranging from nonsmooth to high-order smooth and quasi-self-concordant. To this end, we propose the DADA algorithm, built upon the dual averaging framework and introducing a novel *distance-adaptive* step-size mechanism: the step-size coefficient dynamically depends on both the distance from the current iterate to the initial point and the observed gradient norm, thereby achieving truly parameter-free and precision-agnostic adaptation. We establish theoretical guarantees showing that DADA attains optimal or matching lower-bound convergence rates across all aforementioned convex function classes. Empirical evaluations confirm its robustness and efficiency across diverse benchmark problems.

Technology Category

• Search and Optimization: Non-convex Optimization
• Machine Learning: Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

We present a novel universal gradient method for solving convex optimization problems. Our algorithm -- Dual Averaging with Distance Adaptation (DADA) -- is based on the classical scheme of dual averaging and dynamically adjusts its coefficients based on observed gradients and the distance between iterates and the starting point, eliminating the need for problem-specific parameters. DADA is a universal algorithm that simultaneously works for a broad spectrum of problem classes, provided the local growth of the objective function around its minimizer can be bounded. Particular examples of such problem classes are nonsmooth Lipschitz functions, Lipschitz-smooth functions, H""older-smooth functions, functions with high-order Lipschitz derivative, quasi-self-concordant functions, and $(L_0,L_1)$-smooth functions. Crucially, DADA is applicable to both unconstrained and constrained problems, even when the domain is unbounded, without requiring prior knowledge of the number of iterations or desired accuracy.

Problem

• Adaptive Step Size
• Universal Optimization Method
• Constrained and Unconstrained Problems

Innovation

• DADA
• Dynamic Step Size Adjustment
• Versatile Optimization Algorithm