AdvBDGen: Adversarially Fortified Prompt-Specific Fuzzy Backdoor Generator Against LLM Alignment

2024-10-15

arXiv.org

AI Summary

Reinforcement Learning from Human Feedback (RLHF) alignment poses a risk of covert backdoor injection into large language models (LLMs). Method: This paper proposes a fuzzy trigger mechanism based on prompt-specific paraphrasing—generating variable, semantically preserved rephrasings instead of fixed-token triggers—to enhance backdoor stealth and resilience against data cleansing. We introduce a novel generative-discriminative adversarial framework that jointly optimizes trigger installability, robustness, and cross-model generalizability. Trigger injection requires only 3% fine-tuning data. Contribution/Results: Experiments demonstrate high-success jailbreaking across multiple mainstream LLMs. Our method achieves superior trigger robustness over baselines and exhibits strong resistance to post-hoc mitigation techniques—including output filtering and dataset sanitization—while maintaining semantic fidelity and deployment feasibility.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

With the growing adoption of reinforcement learning with human feedback (RLHF) for aligning large language models (LLMs), the risk of backdoor installation during alignment has increased, leading to unintended and harmful behaviors. Existing backdoor triggers are typically limited to fixed word patterns, making them detectable during data cleaning and easily removable post-poisoning. In this work, we explore the use of prompt-specific paraphrases as backdoor triggers, enhancing their stealth and resistance to removal during LLM alignment. We propose AdvBDGen, an adversarially fortified generative fine-tuning framework that automatically generates prompt-specific backdoors that are effective, stealthy, and transferable across models. AdvBDGen employs a generator-discriminator pair, fortified by an adversary, to ensure the installability and stealthiness of backdoors. It enables the crafting and successful installation of complex triggers using as little as 3% of the fine-tuning data. Once installed, these backdoors can jailbreak LLMs during inference, demonstrate improved stability against perturbations compared to traditional constant triggers, and are more challenging to remove. These findings underscore an urgent need for the research community to develop more robust defenses against adversarial backdoor threats in LLM alignment.

Problem

• Detecting stealthy backdoor triggers in LLM alignment
• Generating prompt-specific backdoors resistant to removal
• Enhancing backdoor stability and transferability across models

Innovation

• Prompt-specific paraphrases as backdoor triggers
• Adversarially fortified generator-discriminator pair
• Effective stealthy transferable backdoor generation