admm algorithm design

This work addresses convex optimization problems involving both smooth and nonsmooth components by proposing the LEAF framework, which introduces scalar-valued Moreau envelope learning into the Alternating Direction Method of Multipliers (ADMM) for the first time. The approach explicitly models the Moreau envelope of the objective function using input-convex neural networks (ICNNs), yielding the MEL-ADMM algorithm and its splitting variant, sMEL-ADMM. These methods substantially reduce model complexity while rigorously preserving convexity and theoretical convergence guarantees. Experimental results demonstrate that the proposed algorithms achieve up to an order-of-magnitude speedup over existing solvers while maintaining a low optimality gap, and exhibit convergence rates comparable to those of classical ADMM.

This work proposes an online learning strategy for accelerating the convergence of the Alternating Direction Method of Multipliers (ADMM) when solving structured convex optimization problems—such as time-varying quadratic programs arising in model predictive control—by dynamically tuning its relaxation parameter. The approach targets scenarios where the problem structure remains fixed but parameters change over time, thereby circumventing the need for costly matrix refactorizations typically required in conventional penalty parameter adjustment. For the first time, convergence guarantees are established for ADMM with time-varying penalty and relaxation parameters. By integrating ideas from reinforcement learning into parameter scheduling, the method achieves substantial improvements in solution efficiency while maintaining low computational overhead. Implemented within the OSQP framework, the proposed strategy significantly reduces both iteration counts and actual solve times on standard quadratic programming benchmarks.

This paper addresses the $ell_0$-regularized optimization problem under general differentiable loss functions. We propose the first branch-and-bound (B&B) framework applicable to arbitrary losses—not restricted to quadratic—overcoming limitations of prior methods relying on “Big-M” or $ell_2$ relaxations. Our key methodological contribution is a unified, flexible relaxation theory encompassing multiple relaxation strategies, enabling closed-form expressions for all critical B&B quantities: dual bounds, branching rules, and node pruning conditions. Based on this theory, we develop El0ps, an open-source solver supporting user-defined losses and regularizers for plug-and-play $ell_0$ modeling. Experiments demonstrate state-of-the-art performance on classical benchmarks; notably, El0ps is the first to provably solve large-scale and non-quadratic $ell_0$ problems, substantially expanding the computational tractability frontier of $ell_0$ optimization.

This work addresses primal-dual optimization in distributed empirical risk minimization, aiming to unify the theoretical understanding of CoCoA and ADMM-type algorithms. Methodologically, we reformulate the dual problem and establish a unified primal-dual update framework. Our key contribution is the first rigorous proof that CoCoA is equivalent to proximal ADMM under a specific choice of the augmented Lagrangian penalty parameter. Furthermore, we demonstrate that judicious tuning of this parameter substantially improves both convergence rate and communication efficiency for various ADMM variants—including consensus, linearized, and proximal ADMM—rendering them uniformly superior to standard CoCoA. We provide a unified convergence analysis with non-asymptotic guarantees. Extensive experiments on synthetic and real-world datasets empirically validate the superiority of parameter-tuned ADMM variants. This work offers new theoretical insights and practical guidance for algorithm selection and design in distributed learning.

This work addresses the lack of provable joint guarantees on generalization and convergence in learned optimization algorithms. Methodologically: (1) it establishes the first PAC-Bayesian generalization bound for unbounded losses, leveraging exponential-family posterior distributions; (2) it formulates optimizer learning as a tractable one-dimensional global optimization problem—convex or non-convex—whose solution is analytically characterizable; and (3) it integrates stochastic optimization design with rigorous theoretical analysis to explicitly trade off convergence rate against generalization error. Empirically, the learned optimizers achieve order-of-magnitude improvements over state-of-the-art methods across four diverse real-world tasks—including neural architecture search, meta-learning, adversarial training, and federated learning—while all gains are underpinned by formal theoretical guarantees. This constitutes the first learning-to-optimize framework endowed with a provably tight PAC-Bayesian generalization bound and jointly certified convergence–generalization performance.

This paper addresses the nonlinear matrix decomposition (NMD) problem: given an input matrix $X$ and target rank $r$, find low-rank factors $W$ and $H$ such that $X approx f(WH)$, where $f$ is an element-wise nonlinearity. We systematically introduce the alternating direction method of multipliers (ADMM) into the NMD framework—enabling support for arbitrary differentiable or nondifferentiable nonlinearities (e.g., ReLU, square, MinMax) and composite loss functions (e.g., least squares, $ell_1$, KL divergence). Theoretically, we establish convergence guarantees for the proposed algorithm under nonconvex settings. Empirically, our method achieves significant improvements in accuracy and generalization across diverse tasks—including sparse nonnegative data fitting, probabilistic circuit modeling, and recommendation systems—while maintaining computational scalability and numerical stability.

This work addresses the computational challenge in traditional alternating direction method of multipliers (ADMM) for bilinear minimax (saddle-point) optimization problems, where evaluating complex proximal operators is often required. The authors propose a novel ADMM variant that decomposes the original problem into two alternating substeps: a generalized projection onto the constraint set \( S \) and a Euclidean projection onto the set \( C \). The key innovation lies in the exact reformulation—without approximation or linearization—of the ADMM proximal operator under the bilinear structure into a computable generalized projection. By integrating tools from convex analysis and projection techniques, the method establishes a provably convergent and computationally efficient framework, significantly simplifying the solution process for bilinear minimax problems.

Existing end-to-end approaches to solving constrained convex optimization problems often fail to strictly satisfy constraints and lack guarantees of optimality. This work proposes a trainable architecture based on unfolded ADMM that enforces hard constraints through an embedded constraint-satisfaction module and a differentiable equality-constraint correction layer, ensuring exact feasibility at every iteration. Furthermore, first-order optimality conditions are incorporated as soft constraints into the training objective to guide convergence toward high-quality solutions. The proposed method uniquely unifies strict constraint satisfaction with optimality-aware learning within an unfolded optimization framework. Empirical results across multiple constrained convex optimization tasks demonstrate substantial improvements over conventional black-box end-to-end models, achieving both high solution accuracy and strong constraint compliance.

This work addresses the critical challenge that modern GPU-accelerated linear programming solvers—such as cuPDLP, which is based on the primal-dual hybrid gradient (PDHG) algorithm—exhibit performance highly sensitive to hyperparameters, yet lack tuning methods with provable generalization guarantees. For the first time, this study establishes structural relationships between hyperparameters and solution trajectories for multiple adaptive techniques in complex first-order LP solvers, including preconditioning, restart strategies, and smoothed weight updates. By integrating convergence analysis of PDHG with a model of structural sensitivity, the authors propose a data-driven hyperparameter learning framework that offers theoretical generalization guarantees under polynomial sample complexity. Experimental results demonstrate that the framework significantly enhances solver efficiency across diverse problem instances.

This work addresses four major challenges in federated learning—functional constraints, communication compression, multi-step local updates, and partial client participation—by proposing a unified optimization framework that avoids projections or dual variables. The method employs a switching gradient mechanism combined with bidirectional error feedback to mitigate compression-induced noise and introduces a soft switching strategy to stabilize updates near the boundary of the feasible region. To the best of our knowledge, this is the first approach to jointly handle all four challenges within a single framework. Theoretically, the algorithm achieves a convergence rate of $O(1/\sqrt{T})$ with high probability. Empirical evaluations on Neyman-Pearson classification and constrained Markov decision process tasks demonstrate its effectiveness and practicality.