Learn the Directions, Normalize the Gains: Post-Training Normalization for LoRA

2026-10-01

AI Summary

This study addresses the issue in LoRA fine-tuning where a few singular directions dominate updates, causing adaptation imbalance and degrading general model capabilities. To mitigate this, it proposes LoRA-Norm, a post-training normalization method that integrates spectral rebalancing with nuclear norm recovery. This approach reveals that balancing adapter gain and equalizing responses constitute independent optimization objectives, enabling gain rebalancing while preserving effective learning directions. Notably, the method requires no additional data or training overhead, achieving optimization at zero inference cost. Extensive evaluations across diverse backbone architectures and tasks demonstrate that LoRA-Norm significantly improves both average specialization and capability retention, outperforming existing spectral pruning and gradient editing techniques.

Abstract

While Low-Rank Adaptation (LoRA) enables efficient task specialization, its learned updates can compromise capabilities beyond the target task. We identify \textbf{adaptation imbalance}: a few singular directions dominate the trained update, leaving its performance sensitive to how gains are allocated. We argue that \textbf{learning where to adapt does not ensure that adaptation gains are well balanced}. This motivates \textbf{LoRA-Norm}, a post-training normalization method that retains learned directions while rebalancing their gains. LoRA-Norm combines spectral rebalancing, a fixed nonlinear transformation of singular values, with nuclear-norm restoration, which preserves the original total spectral mass. It requires no calibration data or additional training and introduces no inference overhead. Across two backbones and three adaptation tasks, LoRA-Norm improves average specialization and capability retention, outperforming the evaluated post-hoc spectral pruning and gradient-guided editing configurations on both measures. Stronger functional equalization brings no consistent additional gains, revealing that balancing adapter gains and equalizing their responses are distinct objectives.

Problem

• Low-Rank Adaptation
• adaptation imbalance
• capability retention
• gain balancing

Innovation

• LoRA-Norm
• post-training normalization
• spectral rebalancing
• nuclear-norm restoration
• adaptation imbalance