Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Models

2024-05-27

arXiv.org

AI Summary

To address the heightened safety risks introduced by LoRA fine-tuning—coupled with the high computational and data requirements of conventional safety-enhancement methods—this paper proposes a training-free, data-free safety augmentation framework. Our approach projects LoRA weights zero-shot onto a safety-aligned subspace jointly defined by the base model and an aligned reference model, enabling safe reparameterization without modifying the original alignment capabilities. To our knowledge, this is the first parameter-efficient, training- and data-agnostic safety reinforcement method that preserves the base model’s alignment performance. Experiments demonstrate that, under mixed benign/malicious fine-tuning, our method improves safety metrics by up to 47% while degrading downstream task accuracy by less than 0.5%; under purely malicious fine-tuning, it maintains safety levels comparable to those of the original aligned model.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Security and Privacy: Large-scale security measurements

Abstract

While large language models (LLMs) such as Llama-2 or GPT-4 have shown impressive zero-shot performance, fine-tuning is still necessary to enhance their performance for customized datasets, domain-specific tasks, or other private needs. However, fine-tuning all parameters of LLMs requires significant hardware resources, which can be impractical for typical users. Therefore, parameter-efficient fine-tuning such as LoRA have emerged, allowing users to fine-tune LLMs without the need for considerable computing resources, with little performance degradation compared to fine-tuning all parameters. Unfortunately, recent studies indicate that fine-tuning can increase the risk to the safety of LLMs, even when data does not contain malicious content. To address this challenge, we propose Safe LoRA, a simple one-liner patch to the original LoRA implementation by introducing the projection of LoRA weights from selected layers to the safety-aligned subspace, effectively reducing the safety risks in LLM fine-tuning while maintaining utility. It is worth noting that Safe LoRA is a training-free and data-free approach, as it only requires the knowledge of the weights from the base and aligned LLMs. Our extensive experiments demonstrate that when fine-tuning on purely malicious data, Safe LoRA retains similar safety performance as the original aligned model. Moreover, when the fine-tuning dataset contains a mixture of both benign and malicious data, Safe LoRA mitigates the negative effect made by malicious data while preserving performance on downstream tasks. Our codes are available at url{https://github.com/IBM/SafeLoRA}.

Problem

• Large Language Model
• Fine-tuning
• Model Security

Innovation

• Safe LoRA
• large language model safety
• harmful data mitigation