Round Trip Translation Defence against Large Language Model Jailbreaking Attacks

2024-02-21

arXiv.org

AI Summary

Large language models (LLMs) are vulnerable to socially engineered jailbreaking attacks—inputs semantically intelligible to humans yet difficult to defend against—where existing defenses achieve sub-50% mitigation rates. To address this, we propose Round-Trip Translation (RTT), a lightweight, transferable, plug-and-play defense that leverages multilingual translation and semantic generalization to reconstruct adversarial prompts. RTT is specifically designed to counter social engineering–based jailbreaks without requiring model fine-tuning, ensuring compatibility across mainstream LLMs. Our approach achieves the first effective mitigation against MathsAttack—reducing its success rate by nearly 40%—and elevates defense against PAIR to over 70%, establishing a new state-of-the-art. RTT demonstrates strong robustness and practical deployability, offering a scalable, architecture-agnostic solution to enhance LLM safety against human-crafted adversarial prompts.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large language models (LLMs) are susceptible to social-engineered attacks that are human-interpretable but require a high level of comprehension for LLMs to counteract. Existing defensive measures can only mitigate less than half of these attacks at most. To address this issue, we propose the Round Trip Translation (RTT) method, the first algorithm specifically designed to defend against social-engineered attacks on LLMs. RTT paraphrases the adversarial prompt and generalizes the idea conveyed, making it easier for LLMs to detect induced harmful behavior. This method is versatile, lightweight, and transferrable to different LLMs. Our defense successfully mitigated over 70% of Prompt Automatic Iterative Refinement (PAIR) attacks, which is currently the most effective defense to the best of our knowledge. We are also the first to attempt mitigating the MathsAttack and reduced its attack success rate by almost 40%. Our code is publicly available at https://github.com/Cancanxxx/Round_Trip_Translation_Defence This version of the article has been accepted for publication, after peer review (when applicable) but is not the Version of Record and does not reflect post-acceptance improvements, or any corrections. The Version of Record is available online at: https://doi.org/10.48550/arXiv.2402.13517 Use of this Accepted Version is subject to the publisher's Accepted Manuscript terms of use https://www.springernature.com/gp/open-research/policies/accepted-manuscript-terms

Problem

• Defending LLMs against social-engineered jailbreaking attacks
• Mitigating adversarial prompts via Round Trip Translation
• Reducing attack success rates of PAIR and MathsAttack

Innovation

• Round Trip Translation defends jailbreaking attacks
• RTT paraphrases and generalizes adversarial prompts
• Versatile, lightweight, transferrable to different LLMs