Stochastic Penalty-Barrier Methods for Constrained Machine Learning

Unknown Date

AI Summary

This work addresses stochastic optimization problems in deep learning that are nonconvex, nonsmooth, and subject to a large number of constraints—such as those arising in fairness-aware training, physics-informed neural networks, and integration of symbolic knowledge. To tackle these challenges, the paper proposes the Stochastic Penalty-Barrier Method (SPBM), which, for the first time, unifies the handling of up to 10,000 constraints within a stochastic, nonconvex, and nonsmooth setting. By leveraging exponential dual averaging, a stabilized penalty scheduling scheme, and Moreau envelope smoothing, SPBM effectively extends classical penalty and barrier methods to this complex regime. The method seamlessly integrates with mainstream optimizers like Adam, incurs only linear computational overhead, and achieves performance on par with or superior to existing constrained optimization baselines across diverse tasks, while offering strong theoretical guarantees of stability and scalability.

Abstract

Constrained machine learning enables fairness-aware training, physics-informed neural networks, and integration of symbolic domain knowledge into statistical models. Despite its practical importance, no general method exists for the non-convex, non-smooth, stochastic setting that arises naturally in deep learning. We propose the Stochastic Penalty-Barrier Method (SPBM), which extends classical penalty and barrier methods to this setting via exponential dual averaging, a stabilized penalty schedule, and the Moreau envelope to handle non-smoothness. Experiments across multiple settings show that SPBM matches or outperforms existing constrained optimization baselines while incurring only linear runtime overhead compared to unconstrained Adam for up to 10,000 constraints.

Problem

• constrained machine learning
• non-convex optimization
• non-smooth optimization
• stochastic optimization

Innovation

• Stochastic Penalty-Barrier Method
• exponential dual averaging
• Moreau envelope
• constrained optimization
• non-smooth non-convex optimization