Distributionally and Adversarially Robust Logistic Regression via Intersecting Wasserstein Balls

2024-07-18

arXiv.org

AI Summary

Adversarial robust optimization (ARO) often suffers from overfitting due to excessive reliance on the empirical data distribution. Method: This paper proposes a unified optimization framework jointly enforcing distributional and adversarial robustness. Its core innovation is the first integration of Wasserstein distributionally robust optimization (DRO) with a data-driven ambiguity set constructed from auxiliary data—yielding a dual-robust model that is both theoretically grounded and computationally tractable. The framework leverages Wasserstein ambiguity set construction, convex reformulation, cross-distribution uncertainty estimation, and an efficient first-order optimization algorithm. Results: Extensive experiments on standard benchmarks demonstrate that the method significantly improves generalization accuracy while preserving—and often enhancing—robustness against adversarial attacks, consistently outperforming state-of-the-art robust baselines.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Search and Optimization: Adversarial Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Adversarially robust optimization (ARO) has become the de facto standard for training models to defend against adversarial attacks during testing. However, despite their robustness, these models often suffer from severe overfitting. To mitigate this issue, several successful approaches have been proposed, including replacing the empirical distribution in training with: (i) a worst-case distribution within an ambiguity set, leading to a distributionally robust (DR) counterpart of ARO; or (ii) a mixture of the empirical distribution with one derived from an auxiliary dataset (e.g., synthetic, external, or out-of-domain). Building on the first approach, we explore the Wasserstein DR counterpart of ARO for logistic regression and show it admits a tractable convex optimization reformulation. Adopting the second approach, we enhance the DR framework by intersecting its ambiguity set with one constructed from an auxiliary dataset, which yields significant improvements when the Wasserstein distance between the data-generating and auxiliary distributions can be estimated. We analyze the resulting optimization problem, develop efficient solutions, and show that our method outperforms benchmark approaches on standard datasets.

Problem

• Address overfitting in adversarially robust logistic regression
• Intersect Wasserstein ambiguity sets for improved robustness
• Develop efficient algorithms for distributionally robust optimization

Innovation

• Intersecting Wasserstein balls for robustness
• Tractable convex optimization reformulation
• Improved performance with auxiliary datasets