Problem
Research questions and friction points this paper is trying to address.
jailbreak defenses
safety-performance trade-offs
over-refusal
inference cost
large language models
When LLM Defenses Backfire: Characterizing Safety, Performance, and Cost Trade-offs
🤖 AI Summary
This study systematically evaluates mainstream jailbreak defense strategies—including rule-based, self-reflective, and multi-turn interactive approaches—across multiple open-source large language models and benchmark datasets, analyzing the trade-offs among safety, utility, and computational efficiency. The work demonstrates that while enhancing security, these defenses often compromise task performance, trigger excessive rejection of benign inputs, or incur substantial inference overhead. Notably, it reveals distinct side-effect patterns across mechanisms: rule-based methods best preserve model utility, conservative self-reflection tends to over-reject legitimate queries, and multi-turn interaction introduces significant computational costs. These findings provide empirical guidance for selecting appropriate defense strategies based on application-specific priorities.
📝 Abstract
Jailbreak defenses are essential for protecting large language models (LLMs), but they can also introduce secondary costs that weaken model utility. We present a systematic study of these defense trade-offs along three dimensions: performance impact, over-refusal on benign inputs, and inference cost. Rather than treating defenses as a single class, we organize them by operational strategy and examine how different strategies correlate with different side-effect profiles. Across state-of-the-art defense methods, widely used benchmark datasets, and representative open-source LLMs, we find that defenses rarely improve downstream capability, but instead vary in how they trade safety gains against usability and efficiency. In particular, rule-based defenses best preserve task performance, highly conservative self-reflective defenses often increase over-refusal, and multi-round defenses incur the largest runtime overhead. These results provide both a benchmark for evaluating defense side effects and practical guidance for selecting defenses under deployment constraints.
Innovation
Methods, ideas, or system contributions that make the work stand out.
LLM defenses
safety-performance trade-offs
over-refusal
inference cost
jailbreak mitigation
🔎 Similar Papers
No similar papers found.
