🤖 AI Summary
This work investigates whether adversarial training in nonlinear models can be reduced to a regularization problem. Focusing on two-layer neural networks, the study provides the first theoretical proof that adversarial risk cannot be equivalent to any weakly data-dependent regularized risk. Empirical evidence further supports this finding on deep architectures such as Wide-ResNet. By integrating theoretical reduction with experimental analysis, the research reveals a fundamental distinction between adversarial robustness and conventional regularization, establishing a clear boundary between the two paradigms. Consequently, it demonstrates that efficient approximation methods proven effective for linear models do not directly generalize to nonlinear neural networks.
📝 Abstract
The vulnerability of ML models to adversarial examples has recently emerged as a major concern. While adversarial training is one of the most effective countermeasures to this issue, its high computational cost remains an obstacle to practical deployment. Recent progress in reducing this cost has relied, in the case of linear models, on a formal equivalence between the adversarial risk and a simpler form of regularized risk. This enabled significantly more efficient training procedures, which naturally raises the question of whether such an equivalence can be extended beyond linear models. In this work, we formally show that no such equivalence is possible for two-layer networks. Our proofs proceed via a reduction to key properties that fundamentally separate the adversarial risk from any simple regularized risk which would only exhibit a weak form of data dependence. Beyond this setting, we provide empirical evidence on Wide-ResNets indicating that the same type of impossibility persists in deeper and more expressive architectures.