Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence

2026-05-29

AI Summary

This work addresses the poor conditioning of the optimization landscape in Low-Rank Adaptation (LoRA), which stems from parameter redundancy and impedes convergence. To mitigate this issue, the authors propose Balanced Low-Rank Adaptation (BaLoRA), a novel method that projects low-rank adaptation factors onto a balanced manifold for the first time. This projection improves the condition number of the loss function while preserving the adapted weights, thereby eliminating the detrimental effects of parameter invariance on optimization. By integrating low-rank adaptation with matrix balancing via manifold projection, BaLoRA significantly accelerates training convergence and consistently outperforms standard LoRA across a range of fine-tuning tasks.

Abstract

Low-Rank Adaptation (LoRA) is the most widely adopted method for fine-tuning large language models. Notably, LoRA is inherently overparameterized: multiple pairs of low-rank factors can yield the same adapted weight matrix. We show--both theoretically and empirically--that these pairs exhibit significantly different condition numbers. As a result, converging to different loss minimizers directly impacts the convergence rate of LoRA. Building on this observation, we introduce Balanced Low-Rank Adaptation (BaLoRA), a variant of LoRA that projects iterates onto a balanced manifold. This manifold improves the conditioning of the loss landscape while preserving the adapted matrix. The projection step is computationally lightweight and integrates seamlessly into existing fine-tuning pipelines. Empirically, BaLoRA converges faster than standard LoRA and achieves superior performance across a range of fine-tuning tasks.

Problem

• LoRA
• parameter invariance
• convergence rate
• condition number
• low-rank adaptation

Innovation

• Balanced LoRA
• parameter invariance
• condition number
• low-rank adaptation
• convergence acceleration