Forewarned is Forearmed: Pre-Synthesizing Jailbreak-like Instructions to Enhance LLM Safety Guardrail to Potential Attacks

2025-08-27

AI Summary

Existing large language models (LLMs) suffer from poor generalization against novel jailbreaking attacks due to distributional shift between safety-aligned training data and real-world adversarial inputs, resulting in a reactive patching paradigm. To address this, we propose IMAGINE, the first framework that synthesizes jailbreak-like instructions iteratively via embedding-space distribution analysis. By modeling the evolutionary dynamics of text generation distributions, IMAGINE dynamically expands the coverage of safety training data, shifting the paradigm from passive mitigation to proactive defense. Extensive experiments on Qwen2.5, Llama3.1, and Llama3.2 demonstrate that IMAGINE significantly reduces success rates across diverse jailbreaking attack families—without compromising the model’s original utility or task performance.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Despite advances in improving large language model(LLM) to refuse to answer malicious instructions, widely used LLMs remain vulnerable to jailbreak attacks where attackers generate instructions with distributions differing from safety alignment corpora. New attacks expose LLMs' inability to recognize unseen malicious instructions, highlighting a critical distributional mismatch between training data and real-world attacks that forces developers into reactive patching cycles. To tackle this challenge, we propose IMAGINE, a synthesis framework that leverages embedding space distribution analysis to generate jailbreak-like instructions. This approach effectively fills the distributional gap between authentic jailbreak patterns and safety alignment corpora. IMAGINE follows an iterative optimization process that dynamically evolves text generation distributions across iterations, thereby augmenting the coverage of safety alignment data distributions through synthesized data examples. Based on the safety-aligned corpus enhanced through IMAGINE, our framework demonstrates significant decreases in attack success rate on Qwen2.5, Llama3.1, and Llama3.2 without compromising their utility.

Problem

• Addressing LLM vulnerability to unseen jailbreak attack instructions
• Bridging distributional gap between training data and real-world attacks
• Reducing attack success rates without compromising model utility

Innovation

• Embedding space analysis generates jailbreak-like instructions
• Iterative optimization evolves text generation distributions
• Augments safety alignment data with synthesized examples