🤖 AI Summary
This work addresses the lack of theoretical convergence guarantees for Low-Rank Adaptation (LoRA) in stochastic optimization, particularly regarding the query complexity required to reach an ε-stationary point. To bridge this gap, the authors introduce LoRA-STORM, the first variance-reduced algorithm tailored for LoRA, incorporating momentum and leveraging a mean-squared smoothness assumption to establish tighter convergence bounds. Under deterministic settings, LoRA-GD achieves a gradient complexity of O(ε⁻⁴). In stochastic settings, the proposed methods LoRA-NSGDM and LoRA-STORM attain stochastic oracle complexities of O(ε⁻⁸) and O(ε⁻⁶), respectively, substantially improving upon existing results in the literature.
📝 Abstract
Low-rank adaptation (LoRA) optimizes $J(B,A)=\mathcal L(W_\mathrm{base}+sBA)$ over two adapters $B \in \mathbb{R}^{m \times r}$ and $A \in \mathbb{R}^{r \times n}$ that form a low-rank update to a frozen pretrained weight matrix $W_\mathrm{base} \in \mathbb{R}^{m \times n}$. The prior analysis shows LoRA-GD takes $\exp\{\mathcal{O}(ε^{-2})\}$ oracle calls to find an $ε$-stationary point such that $\|\nabla J(B,A)\|\leq ε$ in the deterministic setting. We sharpen the analysis and show that $\mathcal{O}(ε^{-4})$ full-gradient evaluations suffice for the same first-order criterion. We further study stochastic LoRA under unbiased gradient estimates and finite variance. We propose LoRA-NSGDM, which finds an $ε$-stationary point with $\mathcal{O}(ε^{-8})$ stochastic oracle complexity. Under the additional mean-square smoothness condition, we use variance reduction strategy and propose LoRA-STORM, which improves the stochastic oracle complexity to $\mathcal{O}(ε^{-6})$.