Private LoRA Fine-tuning of Open-Source LLMs with Homomorphic Encryption

2025-05-12

AI Summary

Sensitive data and gradient leakage pose significant privacy risks during remote fine-tuning of open-source large language models (LLMs). Method: This paper proposes the first interactive private fine-tuning protocol that deeply integrates Low-Rank Adaptation (LoRA) with fully homomorphic encryption (FHE). It decomposes LoRA’s core computations into the encrypted domain and leverages FHE’s native support for quantization and GPU-accelerated ciphertext matrix operations, drastically reducing client-side computational overhead. Contributions/Results: (1) The first end-to-end, FHE-based private fine-tuning framework achieving high efficiency for LoRA; (2) Empirical validation on Llama-3.2-1B demonstrates convergence and practical utility; (3) FHE computation latency is reduced by over 10× compared to CPU-based implementations, enabling low-barrier private fine-tuning on real-world sensitive data—such as medical records and proprietary codebases—without exposing raw data or gradients.

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Security and Privacy: Privacy-enhancing technologies
• Search and Retrieval-Augmented AI: Large language models for search
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Preserving data confidentiality during the fine-tuning of open-source Large Language Models (LLMs) is crucial for sensitive applications. This work introduces an interactive protocol adapting the Low-Rank Adaptation (LoRA) technique for private fine-tuning. Homomorphic Encryption (HE) protects the confidentiality of training data and gradients handled by remote worker nodes performing the bulk of computations involving the base model weights. The data owner orchestrates training, requiring minimal local computing power and memory, thus alleviating the need for expensive client-side GPUs. We demonstrate feasibility by fine-tuning a Llama-3.2-1B model, presenting convergence results using HE-compatible quantization and performance benchmarks for HE computations on GPU hardware. This approach enables applications such as confidential knowledge base question answering, private codebase fine-tuning for AI code assistants, AI agents for drafting emails based on a company's email archive, and adapting models to analyze sensitive legal or healthcare documents.

Problem

• Preserving data confidentiality in LLM fine-tuning
• Private LoRA adaptation using Homomorphic Encryption
• Enabling secure sensitive applications with HE-protected gradients

Innovation

• Homomorphic Encryption secures training data confidentiality
• LoRA technique adapted for private fine-tuning
• Minimal local computing power required for orchestration