No Free Efficiency: Revisiting the Trade-off Between Training Efficiency and Model Vulnerability

2026-09-27

AI Summary

This study investigates whether efficient training strategies compromise model robustness and safety. We conduct the first systematic cross-domain analysis to reveal the intrinsic trade-off between efficiency and vulnerability. By integrating adversarial and privacy attack evaluations, loss landscape analysis, and representational structure assessment, this work establishes a deep connection between internal geometric properties of models and their security risks. Our findings demonstrate that streamlined, efficiency-oriented training variants are more susceptible to attacks, prone to catastrophic forgetting, and exhibit overconfidence. To address these issues, we propose a novel multi-objective optimization paradigm that jointly balances performance, computational cost, and safety. This framework provides both theoretical foundations and practical guidance for the secure and efficient deployment of large-scale models.

Abstract

Training efficiency has become the central driver of recent progress in foundation models. To overcome the massive computational and data requirements of large-scale training, researchers increasingly adopt strategies such as selective data sampling, efficient pre-training, and simplified reinforcement learning pipelines. While these strategies drastically reduce overhead, they prompt a critical, yet neglected question: Is efficiency achieved at the expense of model robustness and security? To our knowledge, we present the first systematic cross-domain investigation of the efficiency-vulnerability trade-off. Across vision and language models, we show that efficiency-oriented training increases susceptibility to adversarial and privacy attacks. We characterize this vulnerability by analyzing the models'internal geometry and functional representations, demonstrating that the evaluated efficient variants consistently exhibit sharper loss geometry together with systematic changes in representational structure. We further extend our analysis to"zero RL training", finding that models trained using simplified RL recipes exhibit substantially greater susceptibility to catastrophic forgetting and more pronounced overconfidence than those trained through conventional alignment pipelines. Our findings suggest that training efficiency is rarely a"free lunch"; rather, the mechanisms that minimize computation can inadvertently compromise safety. We conclude by calling for a paradigm shift toward multi-objective training that jointly optimizes for performance, cost, and security.

Problem

• Training Efficiency
• Model Vulnerability
• Adversarial Robustness
• Foundation Models
• Safety

Innovation

• Training Efficiency
• Model Vulnerability
• Loss Geometry
• Zero RL Training
• Multi-objective Training