Optimization over Trained (and Sparse) Neural Networks: A Surrogate within a Surrogate

2025-05-04

AI Summary

Mixed-integer linear programming (MILP)-based formal verification of neural networks suffers from computational intractability due to inherent nonlinearity and uncertainty in embedded models. Method: This paper proposes a sparse surrogate modeling approach based on neural network pruning: the original network is pruned into a low-precision, highly sparse subnetwork—termed a “surrogate-of-a-surrogate”—which is directly encoded into the MILP framework for adversarial verification. Contribution/Results: We establish, for the first time, that pruned networks with degraded classification accuracy are inherently more amenable to MILP optimization, significantly accelerating adversarial perturbation search. Crucially, no fine-tuning is required. Experiments demonstrate substantial speedups in verification runtime while preserving soundness and completeness. This work introduces an efficient, interpretable paradigm for constrained learning and formal verification of neural networks.

Technology Category

Machine Learning: Adversarial Learning & RobustnessComputer Vision: Adversarial Attacks & RobustnessSearch and Optimization: Adversarial Search

Application Category

Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphsSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingUser Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

We can approximate a constraint or an objective function that is uncertain or nonlinear with a neural network that we embed in the optimization model. This approach, which is known as constraint learning, faces the challenge that optimization models with neural network surrogates are harder to solve. Such difficulties have motivated studies on model reformulation, specialized optimization algorithms, and - to a lesser extent - pruning of the embedded networks. In this work, we double down on the use of surrogates by applying network pruning to produce a surrogate of the neural network itself. In the context of using a Mixed-Integer Linear Programming (MILP) solver to verify neural networks, we obtained faster adversarial perturbations for dense neural networks by using sparse surrogates, especially - and surprisingly - if not taking the time to finetune the sparse network to make up for the loss in accuracy. In other words, we show that a pruned network with bad classification performance can still be a good - and more efficient - surrogate.

Problem

• Optimizing neural networks with embedded surrogates is computationally challenging
• Network pruning improves efficiency without fine-tuning sparse surrogates
• Pruned networks with poor accuracy can still serve as effective surrogates

Innovation

• Neural network pruning for surrogate optimization
• Sparse surrogates speed up MILP verification
• Untuned pruned networks enhance efficiency