Rotated Manifold Optimization for Low-Rank Adaptation

2026-09-28

AI Summary

This study addresses the limitation that existing Low-Rank Adaptation (LoRA) optimizers fail to fully exploit the gauge symmetry inherent in matrix factorization, thereby constraining convergence efficiency and model performance. To overcome this, we propose a novel manifold optimizer incorporating rotational basis normalization, which extends full-parameter matrix optimization to fixed-rank manifolds. By explicitly leveraging the gauge symmetry of low-rank decomposition, our approach efficiently integrates rotation and normalization operations on the manifold to optimize LoRA training. Experimental results demonstrate that the proposed method achieves faster convergence and lower holdout loss, yielding performance superior to or on par with established baselines across both supervised fine-tuning and reinforcement learning tasks.

Abstract

We propose a novel optimizer for low-rank adaptation (LoRA) that explicitly incorporates the gauge symmetry of low-rank factorization. Our optimizer extends recent matrix optimizers for full-parameter training to the manifold of fixed-rank matrices by interpreting them as normalization under a rotated basis. We show how rotation and normalization can be integrated with the fixed-rank manifold efficiently. Our optimizer converges faster to lower held-out loss and achieves better or comparable downstream performance on both supervised finetuning and reinforcement learning tasks.

Problem

• Low-Rank Adaptation
• Manifold Optimization
• Gauge Symmetry
• Fixed-Rank Matrices

Innovation

• Low-Rank Adaptation
• Manifold Optimization
• Gauge Symmetry
• Fixed-Rank Matrices
• Rotated Basis Normalization