Retraction-Free Optimization over the Stiefel Manifold for the LoRA Fine-Tuning

2026-07-28

AI Summary

This work addresses the computational expense of retractions in Stiefel manifold optimization and the sensitivity to hyperparameter tuning in landing-based methods by formulating LoRA fine-tuning as a constrained optimization problem on the Stiefel manifold. The authors propose a retraction-free, penalty-parameter-free direct landing algorithm that leverages the strong convexity of a quadratic penalty function and the proximal smoothness of the manifold to devise an adaptive stepsize strategy. This approach establishes, for the first time, a retraction-free optimization framework with global convergence guarantees. Experimental results demonstrate that the method significantly improves training efficiency on standard benchmarks, achieves state-of-the-art iteration complexity, and delivers superior performance on downstream tasks.

Abstract

Optimization over the Stiefel manifold plays a significant role in various machine learning tasks. Existing methods either use the retraction operators, requiring costly orthonormalization for large-scale matrices, or employ landing methods that rely on careful step size selection and penalty parameter tuning. To address these challenges, we propose a retraction-free and penalty parameter-free algorithm that directly lands on the manifold. By leveraging the strongly-convex-like property of the quadratic penalty function and the proximal smoothness of the Stiefel manifold, we establish global convergence guarantees with the best-known iteration complexities under both constant and diminishing step sizes. Then, we reformulate the low-rank adaptation (LoRA) fine-tuning problem for large language models as a manifold optimization problem, introducing Manifold-LoRA for geometry-accelerated adaptation. This approach employs the proposed landing technique and a carefully designed step size strategy to accelerate the training process. Numerical experiments on benchmark datasets demonstrate the efficiency and strong downstream performance of the proposed method.

Problem

• Stiefel manifold
• retraction-free optimization
• LoRA fine-tuning
• manifold optimization
• orthonormalization

Innovation

• retraction-free optimization
• Stiefel manifold
• LoRA fine-tuning
• manifold landing
• parameter-free algorithm