algorithm unrolling

Deep learning struggles to rigorously incorporate hard constraints due to the lack of differentiable, constraint-aware layers. Method: We propose an end-to-end trainable framework that embeds generic convex optimization problems as differentiable layers within neural networks. We establish the first unified differentiability theory for arbitrary differentiable convex optimization, deriving exact gradients via the implicit function theorem and convex analysis, and extend automatic differentiation to support parameterized convex layers. Unlike prior work restricted to quadratic programming, our approach enables rigorous modeling of linear, semidefinite, and general conic constraints. Contribution/Results: Experiments demonstrate substantial improvements in generalization and constraint satisfaction across control, logical reasoning, and physics-guided learning tasks—effectively bridging a critical gap between convex optimization theory and deep learning practice.

This work proposes a differentiable programming–based framework for learning adaptive optimization algorithms to address the slow convergence and high per-iteration cost of traditional first-order methods in large-scale optimization. By embedding Fenchel–Rockafellar duality theory into automatic differentiation systems, the framework enables end-to-end training and adaptive refinement of duality-driven iterative schemes such as ADMM and PDHG. Implemented uniformly across major deep learning frameworks—including PyTorch, TensorFlow, and JAX—the approach significantly improves both computational efficiency and solution quality on a range of tasks, including linear programming, optimal power flow (OPF), Laplacian regularization, and neural network verification.

Physics-informed neural networks (PINNs) face challenges—including ill-conditioned optimization, slow convergence, and poor generalization—when solving parametric partial differential equations (PDEs). This paper proposes a data-driven neural solver that parameterizes adaptive gradient descent as a neural network, jointly modeling distributions of PDE coefficients and initial/boundary conditions under physical constraints, while dynamically conditioning the optimizer to alleviate loss function ill-conditioning. To our knowledge, this is the first work to introduce neural solvers into parametric PDE settings, enabling end-to-end training via implicit differentiation and backpropagation. Experiments demonstrate a 2–5× speedup in training with enhanced convergence stability. At inference, the solver generalizes robustly to unseen parameter combinations, significantly reducing required iterations while maintaining high accuracy.

This paper addresses non-convex optimization in shallow neural networks across three fundamental tasks: exact representation, function approximation, and regression. We propose a unified convexification framework grounded in mean-field theory. Theoretically, we rigorously prove that the convexified problem admits no relaxation gap and derive an interpretable, closed-form generalization bound that explicitly characterizes hyperparameter influence and provides principled guidelines for optimal selection. Algorithmically, we design a task-adaptive solver: for low-dimensional settings, we employ the simplex method with theoretical guarantees of exact recovery; for high-dimensional settings, we combine sparsification with gradient descent to achieve efficient approximation. Empirical results demonstrate substantial improvements in test performance over standard training heuristics. To our knowledge, this is the first work achieving unified modeling, gap-free convexification, and joint optimization of generalization and algorithmic efficiency across all three tasks.

This work addresses the learning of solution operators for nonlinear partial differential equations (PDEs). Methodologically, it introduces a multilayer perceptron (MLP)-based parametric operator learning framework driven by physical inputs—including boundary conditions, coefficients, and source terms—where a local energy functional, constructed via finite element discretization, is directly adopted as the training loss—a novel formulation. To enhance scalability, the method incorporates element-wise parallelization and randomized sparse grid sampling, significantly improving training efficiency for large-scale problems. Experiments on multiple nonlinear PDE benchmarks demonstrate high-accuracy solution prediction and strong generalization across unseen parameter configurations. Compared to conventional numerical solvers, the approach drastically reduces computational overhead associated with repeated parameter sweeps and enables real-time parametric response prediction.

To address stagnation or divergence of Newton’s method in solving parametric nonlinear systems caused by severe nonlinearity-induced imbalance, this paper proposes a neural operator-preconditioned Newton method. Our approach integrates deep learning priors into classical numerical optimization: (1) we design a fixed-point neural operator (FPNO) that learns, in an end-to-end differentiable manner, the mapping from the current iterate to the true solution—serving as a learned preconditioner; and (2) we introduce an adaptive negative-step mechanism that relaxes reliance on local convexity assumptions inherent in conventional line search and trust-region methods. The method significantly enhances convergence robustness and computational efficiency for strongly nonlinear, multiscale parametric problems. Experiments across multiple real-world physical modeling tasks demonstrate 2–5× speedup over classical solvers, with markedly reduced sensitivity to initial guesses.

This work proposes a minimal neural architecture based on Horner-factorized polynomials to address the challenge of large parameter counts in neural networks for solving differential equations, which hinders deployment in resource-constrained settings. The method constructs an implicit, differentiable trial solution with only a few learnable coefficients that inherently satisfies initial conditions. By integrating a piecewise-continuous (spline-like) stitching strategy with a residual collocation scheme, the approach achieves high accuracy and smoothness in both the solution and its derivatives while drastically reducing model size. Experiments on ordinary differential equations and the heat equation demonstrate that the proposed method accurately approximates the true solutions and their derivatives using merely dozens of parameters, outperforming comparably sized small MLPs and sinusoidal representation networks.

To address the accumulation of autoregressive errors—severely amplified in long-term rollouts of chaotic systems—caused by directly applying neural corrections to state variables in hybrid PDE solvers, this paper proposes an **indirect correction mechanism**: learned correction terms are embedded into the governing equations rather than applied to numerical solutions. This approach suppresses error propagation at the modeling level; theoretical analysis shows it reduces the error amplification rate to $O(Delta t^{-1} + L)$. The method is solver- and architecture-agnostic. Leveraging numerical differentiation, automatic differentiation, and end-to-end differentiable modeling, we validate it across multiscale physical systems—from 1D chaotic dynamics to 3D turbulent flows. Results demonstrate up to a 158.7% improvement in R², effective suppression of coarse-graining-induced blow-ups, and speedups of several orders of magnitude in 3D turbulence simulation—achieving high efficiency, strong stability, and physical consistency.

Differentiable programming struggles with discrete control structures—such as conditionals and loops—due to the absence of classical derivatives, often excluding iterative logic from gradient-based learning and severely limiting neural networks’ capacity to model algorithmic tasks. This paper introduces Cajal, a typed, higher-order language grounded in linear logic and recursive types, which for the first time compiles iterative, semantically well-defined programs into *linear recurrent neurons* that are behaviorally equivalent to their source programs. Crucially, this compilation is underpinned by a constructive semantic mapping that guarantees correctness and enables seamless integration of iterative control flow into gradient optimization. Experiments on iterative image transformation tasks demonstrate that models incorporating these neurons achieve faster convergence and markedly improved data efficiency, empirically validating the effectiveness and practicality of co-modeling discrete structures and deep learning.

This work addresses the high computational cost and slow training of conventional coordinate-based neural networks for solving differential equations, as well as the limited accuracy of existing grid-based methods in computing high-order derivatives due to their reliance on linear interpolation. The authors propose a novel representation that integrates differentiable feature grids with infinitely differentiable radial basis function (RBF) interpolation—introducing RBFs into feature grids for the first time to enable accurate high-order derivative computation. A multi-resolution collocated grid structure is designed to efficiently capture high-frequency details while preserving global gradient stability. The model is trained implicitly using the governing differential equation as the loss function. On Poisson, Helmholtz, and Kirchhoff–Love thin plate problems, it achieves 5–20× speedups over MLP-based approaches (reducing solve times from minutes to seconds) while maintaining comparable accuracy and a compact model size.