Generalist++: A Meta-learning Framework for Mitigating Trade-off in Adversarial Training

2025-10-15

AI Summary

Adversarial training (AT) suffers from degraded natural accuracy and poor cross-attack robustness generalization. To address these challenges, we propose a multi-task collaborative generalization framework that decomposes global robust learning into multiple specialized subtasks, each optimized by a dedicated base learner. Knowledge fusion and co-evolution between base learners and the global model are achieved via dynamic parameter interpolation, gradient synchronization, and periodic task reallocation—without incurring additional inference overhead. The framework supports three lightweight variants for diverse deployment scenarios. Theoretical analysis shows that our method reduces generalization error. Empirically, it achieves average improvements of 2.1% in natural accuracy and 3.7% in robust accuracy across multiple adversarial attacks, significantly outperforming baselines including PGD-AT and TRADES. This work advances the development of general-purpose robust classifiers.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Despite the rapid progress of neural networks, they remain highly vulnerable to adversarial examples, for which adversarial training (AT) is currently the most effective defense. While AT has been extensively studied, its practical applications expose two major limitations: natural accuracy tends to degrade significantly compared with standard training, and robustness does not transfer well across attacks crafted under different norm constraints. Unlike prior works that attempt to address only one issue within a single network, we propose to partition the overall generalization goal into multiple sub-tasks, each assigned to a dedicated base learner. By specializing in its designated objective, each base learner quickly becomes an expert in its field. In the later stages of training, we interpolate their parameters to form a knowledgeable global learner, while periodically redistributing the global parameters back to the base learners to prevent their optimization trajectories from drifting too far from the shared target. We term this framework Generalist and introduce three variants tailored to different application scenarios. Both theoretical analysis and extensive experiments demonstrate that Generalist achieves lower generalization error and significantly alleviates the trade-off problems compared with baseline methods. Our results suggest that Generalist provides a promising step toward developing fully robust classifiers in the future.

Problem

• Mitigating natural accuracy degradation in adversarial training
• Improving robustness transfer across different attack types
• Addressing trade-offs between accuracy and adversarial robustness

Innovation

• Partitions generalization goal into specialized base learners
• Interpolates parameters to form knowledgeable global learner
• Periodically redistributes global parameters to base learners