differentiable optimization

Classical $L$-smoothness-based convergence analyses fail for nonsmooth neural networks (e.g., ReLU networks), leading to systematic theoretical misjudgments in nondifferentiable settings. Method: The authors employ nonsmooth optimization theory, Lipschitz analysis, and comparative convergence studies to rigorously characterize the behavior of normalized gradient descent methods (NGDMs). Contributions/Results: They establish that NGDMs exhibit fundamentally distinct convergence dynamics compared to standard gradient descent; reveal that $L_1$ regularization can counterintuitively increase parameter magnitudes—undermining pruning efficacy—in nonsmooth regimes; extend the “Edge of Stability” phenomenon to nonconvex, nonsmooth functions for the first time; and disprove the implicit assumption that algorithms like RMSProp behave identically in differentiable versus nondifferentiable settings. Collectively, these findings challenge the prevailing smoothness-dependent paradigm in deep learning optimization and lay the groundwork for new theoretical frameworks tailored to realistic neural network architectures.

Existing differentiable quadratic programming (QP) methods rely on solver-specific implementations, hindering seamless integration into neural networks or bilevel optimization pipelines and restricting solver choice. This paper introduces dQP—the first explicit differentiation framework based on the active set, enabling end-to-end differentiability for arbitrary black-box QP solvers (compatible with 15+ mainstream solvers) without modifying solver source code; only the optimal solution and active constraint set are required for backward propagation. dQP unifies convex optimization theory, implicit function differentiation, and automatic differentiation, supporting both dense small-scale and sparse large-scale QP problems. Experiments demonstrate that dQP consistently outperforms prior differentiable QP methods across diverse benchmarks. Moreover, it enables a novel bilevel geometric optimization task, showcasing broad applicability beyond standard QP settings.

There exists a significant gap between the theoretical convergence guarantees of deep learning optimization algorithms and their empirical performance, largely due to commonly adopted assumptions—such as Hessian boundedness—that lack empirical validation. Method: We introduce the first trajectory-aware measurement framework tightly aligned with key theoretical quantities, systematically evaluating the validity of mainstream assumptions across diverse architectures and datasets using large-scale training runs. Our framework quantifies dynamic properties—including gradient norms, Hessian spectral characteristics, and loss curvature—along optimization trajectories. Contribution/Results: We find that all examined theoretical assumptions fail to reliably predict actual convergence behavior and exhibit no robust correlation with optimization performance. This work uncovers a fundamental misalignment between theoretical modeling and practice, establishing the first reproducible benchmark for empirically calibrating and reconstructing optimization theory.

This work addresses the slow convergence of gradient descent on complex objectives and its reliance on strong global smoothness assumptions. We introduce *directional smoothness*, a novel geometric concept characterizing the local smoothness of the objective function along the optimization trajectory—thereby circumventing restrictive global Lipschitz continuity requirements. Leveraging this path-dependent characterization, we derive a trajectory-aware suboptimality bound and formulate an implicit adaptive step-size equation. We theoretically establish that Polyak’s step size and normalized gradient descent inherently achieve path-adaptive fast convergence. Our methodology integrates directional smoothness analysis, implicit step-size design, and convergence theory for both convex and nonconvex settings. Experiments on logistic regression demonstrate that our new bound substantially improves upon classical $L$-smoothness-based guarantees. Notably, this is the first work to provide path-dependent convergence rates for these two canonical algorithms without requiring prior knowledge of smoothness parameters.

Differentiable optimization in robot visual localization often suffers from local minima and gradient distortion—especially in low-light keypoint detection—compromising robustness and accuracy. Method: This paper proposes a certifiably differentiable framework based on polynomial optimization (POP), which reformulates POP problems into semidefinite programming (SDP) relaxations with certified backward propagation. It integrates implicit differentiation with PyTorch-based end-to-end training, ensuring global optimality guarantees while maintaining computational efficiency. Contribution/Results: To the best of our knowledge, this is the first work to enable certified backpropagation through SDP relaxations of POP problems, theoretically guaranteeing gradient correctness. Experiments demonstrate that the method substantially mitigates failure modes of mainstream differentiable optimizers, significantly improving both keypoint detection robustness and localization accuracy under low-light conditions in robotic visual localization tasks.

This work proposes a differentiable programming–based framework for learning adaptive optimization algorithms to address the slow convergence and high per-iteration cost of traditional first-order methods in large-scale optimization. By embedding Fenchel–Rockafellar duality theory into automatic differentiation systems, the framework enables end-to-end training and adaptive refinement of duality-driven iterative schemes such as ADMM and PDHG. Implemented uniformly across major deep learning frameworks—including PyTorch, TensorFlow, and JAX—the approach significantly improves both computational efficiency and solution quality on a range of tasks, including linear programming, optimal power flow (OPF), Laplacian regularization, and neural network verification.

This work addresses the slow convergence of gradient descent on high-frequency targets in over-parameterized neural networks—a phenomenon attributed to spectral bias—and proposes a regularized Newton method to overcome this limitation. The authors introduce the “Neural Newton Tangent Kernel” (NNTK) to characterize the training dynamics of the proposed method in the infinite-width limit. By analyzing the spectral properties of the NNTK, they demonstrate that the regularization parameter uniformly controls the lower bound of its eigenvalues, thereby mitigating spectral bias. They further establish scaling rules for the regularization with respect to network width, ensuring a positive-definite Hessian and linearized training behavior. Theoretically, the method achieves global exponential convergence to a zero-loss solution for both low- and high-frequency targets in sufficiently wide networks, significantly outperforming standard gradient descent.

This work addresses the universal approximation problem for differentiable mappings and their derivatives on infinite-dimensional weighted manifolds. By introducing a weighted Nachbin theorem, it overcomes the classical restriction to compact sets and extends the approximation capability of function-input neural networks from function values to their derivatives. The approach integrates Banach space linear readouts, horizontal and vertical derivatives, and path signature techniques to establish the first universal approximation theorem that simultaneously approximates both a functional and its derivative on non-anticipative functionals and path-space functionals. Theoretically, it is shown that linear functionals of path signatures can effectively approximate such functionals along with their directional derivatives, substantially enhancing the expressive power and applicability of infinite-dimensional neural networks.

This work addresses the challenge of characterizing the highly complex loss landscape in large language model (LLM) pretraining, where existing theories struggle to balance analytical tractability with accurate dynamic prediction. By performing Taylor expansions of both the model and loss function at mid-training, the authors construct a local quadratic approximation and combine it with Lanczos quadrature and Hessian spectral estimation. For the first time, they validate this approach on a 150M-parameter LLM trained on 3B tokens, demonstrating predictive accuracy over a training window spanning 10% of total steps. Their analysis reveals that the quadratic model faithfully captures optimization trajectories, that the tail structure of the Hessian spectrum is strongly influenced by batch size, preconditioning, and training stage, and that optimization typically resides in a stochastic edge-of-stability regime dictated by batch size—uncovering a deep connection between local stability and hyperparameter choice.

This work addresses the challenge of efficiently and accurately solving parameterized problems—such as parametric ordinary differential equations and inverse problems with wave constraints—in severely ill-posed regimes. The authors propose a gradient flow–based neural network training method that integrates analytic activation functions with a residual network architecture, reformulating parameter learning as the solution of an ordinary differential equation system. By leveraging Łojasiewicz inequality theory, the approach provides rigorous guarantees of convergence during training. Experimental results demonstrate that the method effectively captures the dependency structure of parametric ODE solutions and yields reasonable approximations to inverse problems even under extreme ill-posedness, offering a stable and computationally feasible framework for tackling complex parameterized systems.