Stand on The Shoulders of Giants: Building JailExpert from Previous Attack Experience

2025-08-25

AI Summary

Existing jailbreaking attacks against large language models (LLMs) suffer from poor reusability of historical attack experiences and rapid obsolescence of templates due to model updates. Method: This paper proposes the first automated jailbreaking framework supporting structured modeling and dynamic evolution of attack experiences. It introduces formal representations of attack knowledge, a semantic-drift-aware hierarchical clustering mechanism for experience organization, and integrates dynamic prompt generation with iterative optimization to enable cross-model transfer under black-box settings. Contribution/Results: The core innovation lies in the continuously evolving experience pool, which mitigates redundant exploration and enhances generalization. Experiments demonstrate a 17% average improvement in attack success rate and a 2.7× speedup in efficiency over state-of-the-art methods. The framework’s effectiveness and scalability are validated across multiple LLMs and diverse adversarial scenarios.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large language models (LLMs) generate human-aligned content under certain safety constraints. However, the current known technique ``jailbreak prompt'' can circumvent safety-aligned measures and induce LLMs to output malicious content. Research on Jailbreaking can help identify vulnerabilities in LLMs and guide the development of robust security frameworks. To circumvent the issue of attack templates becoming obsolete as models evolve, existing methods adopt iterative mutation and dynamic optimization to facilitate more automated jailbreak attacks. However, these methods face two challenges: inefficiency and repetitive optimization, as they overlook the value of past attack experiences. To better integrate past attack experiences to assist current jailbreak attempts, we propose the extbf{JailExpert}, an automated jailbreak framework, which is the first to achieve a formal representation of experience structure, group experiences based on semantic drift, and support the dynamic updating of the experience pool. Extensive experiments demonstrate that JailExpert significantly improves both attack effectiveness and efficiency. Compared to the current state-of-the-art black-box jailbreak methods, JailExpert achieves an average increase of 17% in attack success rate and 2.7 times improvement in attack efficiency. Our implementation is available at href{https://github.com/xiZAIzai/JailExpert}{XiZaiZai/JailExpert}

Problem

• Automating jailbreak attacks on LLMs using past experiences
• Addressing inefficiency and repetition in current jailbreak methods
• Formally representing and dynamically updating attack experience structure

Innovation

• Formal representation of experience structure
• Group experiences based on semantic drift
• Dynamic updating of experience pool