A Fully Parameter-Free Second-Order Algorithm for Convex-Concave Minimax Problems with Optimal Iteration Complexity

2024-07-04

arXiv.org

AI Summary

This paper studies convex-concave minimax optimization problems and proposes FF-CR, the first fully parameter-free second-order algorithm: it requires no prior knowledge of Lipschitz constants, bounds on the initial distance to a solution, or other problem-specific parameters. Built upon the cubic regularization framework, FF-CR integrates adaptive step sizes with stopping criteria driven by gradient norms and duality gaps, thereby eliminating all dependence on unknown problem constants. In terms of gradient norm, FF-CR achieves an iteration complexity of $Oig(( ho |z^0 - z^*|^2 / varepsilon)^{2/3}ig)$ to attain an $varepsilon$-optimal solution—matching the theoretical lower bound. Numerical experiments demonstrate that FF-CR is both highly efficient and robust across diverse problem instances, substantially broadening the practical applicability of second-order minimax optimization methods.

Technology Category

• Search and Optimization: Non-convex Optimization
• Computer Vision: Learning & Optimization for CV
• Machine Learning: Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Security and Privacy: Applications of cryptography

Abstract

In this paper, we study second-order algorithms for the convex-concave minimax problem, which has attracted much attention in many fields such as machine learning in recent years. We propose a Lipschitz-free cubic regularization (LF-CR) algorithm for solving the convex-concave minimax optimization problem without knowing the Lipschitz constant. It can be shown that the iteration complexity of the LF-CR algorithm to obtain an $epsilon$-optimal solution with respect to the restricted primal-dual gap is upper bounded by $mathcal{O}(frac{ ho|z^0-z^*|^3}{epsilon})^{frac{2}{3}}$, where $z^0=(x^0,y^0)$ is a pair of initial points, $z^*=(x^*,y^*)$ is a pair of optimal solutions, and $ ho$ is the Lipschitz constant. We further propose a fully parameter-free cubic regularization (FF-CR) algorithm that does not require any parameters of the problem, including the Lipschitz constant and the upper bound of the distance from the initial point to the optimal solution. We also prove that the iteration complexity of the FF-CR algorithm to obtain an $epsilon$-optimal solution with respect to the gradient norm is upper bounded by $mathcal{O}(frac{ ho|z^0-z^*|^2}{epsilon})^{frac{2}{3}}$. Numerical experiments show the efficiency of both algorithms. To the best of our knowledge, the proposed FF-CR algorithm is the first completely parameter-free second-order algorithm for solving convex-concave minimax optimization problems, and its iteration complexity is consistent with the optimal iteration complexity lower bound of existing second-order algorithms with parameters for solving convex-concave minimax problems.

Problem

• Solving convex-concave minimax problems without Lipschitz constant knowledge
• Developing parameter-free second-order algorithms for minimax optimization
• Achieving optimal iteration complexity for gradient norm termination

Innovation

• Lipschitz-free cubic regularization algorithm for minimax
• Fully parameter-free cubic regularization without constants
• Best iteration complexity for gradient norm termination