lipschitz-constrained learning

This work addresses the challenge of verifying Lipschitz constants in conventional neural networks, which typically relies on computationally expensive methods or overly loose trivial bounds that fail to guarantee adversarial robustness and generalization. The authors propose a novel “verification-by-training” paradigm that integrates structural design to directly optimize and tighten trivial Lipschitz bounds during training, thereby circumventing complex post-hoc verification. Key innovations include norm-saturating polynomial activations (polyactivations), unbiased sinusoidal layers, and extensions to non-Euclidean norms, collectively eliminating three major sources of bound looseness. On MNIST, the resulting networks achieve Lipschitz bounds several orders of magnitude lower than existing approaches, with less than 10% error relative to the true Lipschitz constant, significantly enhancing both robustness and generalization performance.

Deep neural networks suffer from stability issues, including drastic prediction fluctuations under minor input/parameter perturbations and optimization difficulties induced by sharp loss landscapes. To address these challenges, this paper proposes a unified multi-perspective stability optimization framework. Our method jointly integrates Lipschitz continuity constraints, randomized smoothing, and loss-function curvature regularization. We design a differentiable Lipschitz-constrained layer and an efficient spectral norm computation algorithm, and establish a model robustness certification mechanism grounded in the Lipschitz constant. Experimental results demonstrate that our approach significantly enhances adversarial robustness and generalization performance, alleviates optimization instability, yields smoother loss landscapes, and provides verifiable stability guarantees—thereby bridging theoretical robustness certification with practical training efficacy.

This work addresses the insufficient certified robustness of deep classifiers by proposing a novel framework that jointly optimizes the decision boundary margin in output space and the Lipschitz constant along vulnerable input directions. Methodologically, it introduces (1) a dynamic margin maximization mechanism that explicitly enlarges inter-class safety margins in the logit space; (2) a differentiable, tight upper bound estimator for the Lipschitz constant, incorporating both activation monotonicity and Lipschitz continuity constraints; and (3) a new differentiable activation layer designed specifically for robustness. Vulnerability-aware regularization enables end-to-end training. Extensive experiments on MNIST, CIFAR-10, and Tiny-ImageNet demonstrate substantial improvements in certified accuracy and generalization performance, consistently outperforming state-of-the-art certified robustness methods.

This work addresses the neglect of input-space regularity mechanisms in theoretical analyses of deep neural network generalization error bounds, focusing specifically on the dynamic evolution of the empirical Lipschitz constant during the double-descent phenomenon. Methodologically, we conduct a systematic analysis of SGD training trajectories, gradient magnitude estimation, loss landscape curvature approximation, and phase segmentation of double descent. Our key contribution is the first empirical demonstration that the Lipschitz constant exhibits pronounced non-monotonic surge-and-decay behavior near the critical transition regime—precisely synchronized with peaks and troughs in test error. Furthermore, we establish that, near the critical point, the norm of parameter-space gradients tightly couples with the input-space Lipschitz constant; moreover, both model complexity and optimization dynamics are jointly characterized by loss curvature and the Euclidean distance of parameters from initialization. This work provides a novel geometric perspective and quantifiable mechanistic framework for understanding double descent.

Traditional conformal prediction (CP) fails under adversarial attacks, while existing robust CP methods suffer from excessively large prediction sets or high computational overhead on large-scale tasks. To address this, we propose Lip-RCP—the first efficient robust prediction framework that deeply integrates 1-Lipschitz robust neural networks with CP. Methodologically, we impose Lipschitz constraints to ensure output stability and derive, for the first time, a theoretical worst-case coverage bound for standard CP under arbitrary attack magnitudes. Experiments on medium- and large-scale benchmarks (e.g., ImageNet) show that Lip-RCP reduces robust prediction set size by up to 42% over state-of-the-art methods while accelerating inference by 3.8×. Crucially, it strictly guarantees both nominal coverage ≥90% and finite-sample robust coverage—without compromising statistical validity.

Existing methods struggle to simultaneously achieve high accuracy, robustness, and calibration in neural networks. This work proposes Lipschitz Scaling Training (LiST), which establishes, for the first time, a theoretical connection between Lipschitz constraints and temperature scaling. By dynamically adjusting the global Lipschitz constant during training, LiST embeds calibration directly into the learning process, automatically identifying a calibration-optimal operating point along the accuracy–robustness Pareto frontier. The method integrates margin-aware Lipschitz constraints, dynamic constant adaptation, and calibration-aware optimization, and further improves sample efficiency by reusing calibration data after convergence. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate that LiST matches baseline performance in both accuracy and robustness while achieving well-calibrated predictions without any post-hoc processing.

This work establishes convergence guarantees for gradient descent in feedforward neural networks of arbitrary width and depth, without requiring special initialization schemes or data assumptions. By introducing a generalized Lipschitz smoothness condition, the authors characterize the structural properties of the composition between common activation functions—such as tanh and sigmoid—and the mean squared error loss, showing that multi-layer compositions partially retain Lipschitz continuity. Leveraging parameter norm control and a descent lemma, the theoretical analysis demonstrates that for an L-layer network, the minimum gradient norm over T iterations converges to zero at a rate of O(1/T^{1/L}). This result provides the first characterization of the global convergence rate of gradient descent for deep networks under general conditions.

This work addresses the insufficient robustness of shallow neural networks under adversarial attacks by proposing a convex-constrained optimization-based post-processing method that efficiently computes the global optimum of the Lipschitz-regularized training objective. Starting from a pre-trained network as the initialization point, the approach strictly enforces Lipschitz constraints while preserving or even enhancing the model’s original performance. Experimental results across multiple real-world regression datasets demonstrate that the resulting models achieve substantially lower Lipschitz-regularized loss and, on several datasets, simultaneously attain higher accuracy and stronger adversarial robustness. These findings validate the effectiveness and superiority of the proposed method as a general-purpose post-processing step for improving both stability and performance of neural networks.

Neural networks often struggle to strictly satisfy nonlinear constraints during inference, which hinders their deployment in safety-critical applications. This work proposes HardNet++, the first method capable of enforcing hard satisfaction of general nonlinear equality and inequality constraints, overcoming the limitation of existing approaches that are restricted to specific constraint forms. By integrating damped local linearization, differentiable projection layers, and end-to-end training, HardNet++ guarantees constraint compliance simultaneously during both training and inference. Evaluated on model predictive control tasks, HardNet++ achieves high-precision constraint adherence while preserving solution optimality.