AlignGuard-LoRA: Alignment-Preserving Fine-Tuning via Fisher-Guided Decomposition and Riemannian-Geodesic Collision Regularization

2025-08-04

AI Summary

LoRA fine-tuning often induces alignment drift, compromising model safety and behavioral constraints. To address this, we propose AlignLoRA: a framework that enforces safe alignment preservation via Fisher-guided subspace constraints and geodesic separation regularization, imposing Riemannian overlap penalties on the low-rank update manifold. We further introduce DriftCaps, a benchmark to quantify safety degradation. AlignLoRA integrates Riemannian geometric analysis with task-aware stability mechanisms—requiring no additional data or inference overhead. Empirical evaluation across multiple safety-critical benchmarks demonstrates up to 50% reduction in alignment drift while preserving downstream task performance. Moreover, we present the first empirical validation of scaling laws for catastrophic forgetting in alignment contexts and show that AlignLoRA significantly mitigates post-fine-tuning loss surges.

Technology Category

• Natural Language Processing: Safety and Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Low-rank adaptation (LoRA) has become a standard tool for efficiently fine-tuning large language models (LLMs). Yet, even minor LoRA updates can induce alignment drift, weakening safety and behavioral constraints through entangled parameter changes. To address this, we propose AlignGuard-LoRA (AGL), a principled framework for preserving alignment during finetuning. AGL introduces several key components: a primary task loss for supervision, Fisher Information Matrix-based regularization to restrict updates in alignment-sensitive subspaces, and task-specific regularization to stabilize the integration of new knowledge. We further introduce collision-aware regularization, blending Riemannian overlap -- which penalizes coordinate-wise interference -- and geodesic separation -- which encourages disjoint update geometry. We curate DriftCaps, a targeted diagnostic benchmark of safe and unsafe prompts designed to quantify alignment drift and safety degradation. Empirical evaluations show that AGL mitigates alignment drift by up to 50% on safety-critical benchmarks without degrading downstream task performance. Comprehensive ablation confirms that each component contributes distinctly to preserving latent safety behaviors. Finally, we derive and validate a scaling law for catastrophic forgetting, revealing that AGL flattens post-finetuning loss escalation while preserving adaptation dynamics. AGL is a structurally grounded refinement of LoRA, ensuring alignment preservation with minimal trade-offs. To encourage further exploration and development, we open-source our implementation.

Problem

• Prevent alignment drift in LoRA fine-tuning of LLMs
• Mitigate safety degradation via Fisher-guided decomposition
• Stabilize new knowledge integration with collision-aware regularization

Innovation

• Fisher Information Matrix-based regularization for alignment-sensitive subspaces
• Collision-aware regularization with Riemannian and geodesic techniques
• Diagnostic benchmark DriftCaps for alignment drift quantification