Distributional Adversarial Loss

2024-06-05

arXiv.org

AI Summary

Conventional adversarial robustness research models perturbations as pointwise sets, failing to capture the inherent stochasticity in real-world defenses such as randomized smoothing and input discretization. Method: We propose **distributional adversarial loss**, modeling allowable perturbations for each sample as an input-dependent family of probability distributions (e.g., those induced by randomized smoothing or discretization), and defining the robust optimization objective as the worst-case expected loss over this distribution family. We unify randomized smoothing and input discretization under a single theoretical framework; derive generalization bounds based on VC dimension and distribution-family complexity; and introduce a generic derandomization procedure that preserves theoretical robustness guarantees while enhancing empirical resilience. Contribution/Results: Experiments demonstrate that derandomized models achieve significantly improved robustness against diverse white-box and black-box attacks, validating both the theoretical formulation and practical efficacy of our approach.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

A major challenge in defending against adversarial attacks is the enormous space of possible attacks that even a simple adversary might perform. To address this, prior work has proposed a variety of defenses that effectively reduce the size of this space. These include randomized smoothing methods that add noise to the input to take away some of the adversary's impact. Another approach is input discretization which limits the adversary's possible number of actions. Motivated by these two approaches, we introduce a new notion of adversarial loss which we call distributional adversarial loss, to unify these two forms of effectively weakening an adversary. In this notion, we assume for each original example, the allowed adversarial perturbation set is a family of distributions (e.g., induced by a smoothing procedure), and the adversarial loss over each example is the maximum loss over all the associated distributions. The goal is to minimize the overall adversarial loss. We show generalization guarantees for our notion of adversarial loss in terms of the VC-dimension of the hypothesis class and the size of the set of allowed adversarial distributions associated with each input. We also investigate the role of randomness in achieving robustness against adversarial attacks in the methods described above. We show a general derandomization technique that preserves the extent of a randomized classifier's robustness against adversarial attacks. We corroborate the procedure experimentally via derandomizing the Random Projection Filters framework of cite{dong2023adversarial}. Our procedure also improves the robustness of the model against various adversarial attacks.

Problem

• Defines distributional adversarial loss using perturbation distributions
• Unifies randomized smoothing and robust learning in PAC-setting
• Investigates derandomization for preserving adversarial robustness

Innovation

• Introduces distributional adversarial loss concept
• Unifies randomized smoothing and robust learning
• Proposes derandomization preserving robustness