PoLAR: Polar-Decomposed Low-Rank Adapter Representation

2025-06-03

AI Summary

Low-rank adaptation (LoRA) of large language models suffers from low subspace utilization and suboptimal fine-tuning performance due to the significant gap between stable rank and algebraic rank in learned low-rank updates. Method: This work introduces polar decomposition into LoRA parameterization for the first time, factorizing the low-rank update into two orthogonal matrices and a diagonal scaling matrix, with constraints imposed on the Stiefel manifold. Contribution/Results: We theoretically prove that this design enables exponential acceleration in convergence rate. Coupled with Riemannian optimization, it substantially improves subspace utilization efficiency. Empirical evaluation across three major benchmarks—language understanding, commonsense reasoning, and mathematical problem solving—demonstrates consistent and significant performance gains on base models ranging from 350M to 27B parameters, validating both generalizability and effectiveness.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We show that low-rank adaptation of large-scale models suffers from a low stable rank that is well below the linear algebraic rank of the subspace, degrading fine-tuning performance. To mitigate the underutilization of the allocated subspace, we propose PoLAR, a parameterization inspired by the polar decomposition that factorizes the low-rank update into two direction matrices constrained to Stiefel manifolds and an unconstrained scale matrix. Our theory shows that PoLAR yields an exponentially faster convergence rate on a canonical low-rank adaptation problem. Pairing the parameterization with Riemannian optimization leads to consistent gains on three different benchmarks testing general language understanding, commonsense reasoning, and mathematical problem solving with base model sizes ranging from 350M to 27B.

Problem

• Low-rank adaptation suffers from low stable rank
• Proposes PoLAR to better utilize subspace via polar decomposition
• Achieves faster convergence and gains on multiple benchmarks

Innovation

• Polar decomposition for low-rank adaptation
• Stiefel manifold-constrained direction matrices
• Riemannian optimization for faster convergence