lora fine-tuning

LoRA fine-tuning suffers from imbalanced parameter updates and slow convergence due to the non-uniqueness of low-rank decomposition. To address this, we propose Stepwise Optimal Low-Rank Reconstruction (SOLR): at each training step, SOLR dynamically re-decomposes the low-rank adaptation matrices based on gradient analysis, optimizing toward minimization of a theoretically grounded loss upper bound. This yields balanced parameter updates and flattens the loss landscape. SOLR requires no architectural modifications and is plug-and-play compatible with mainstream LLMs (e.g., DeBERTaV3, LLaMA). On NLU and commonsense reasoning benchmarks, SOLR accelerates convergence by 22–35% over LoRA and QLoRA while significantly improving generalization; its computational overhead is negligible. Our key contribution is the first integration of real-time low-rank reconstruction with loss upper-bound-driven optimization into the adapter-based fine-tuning paradigm.

To address high parameter redundancy, substantial computational overhead, and cross-layer parameter interference in LoRA-based fine-tuning of large language models, this paper proposes EffiLoRA. Methodologically, EffiLoRA is the first to jointly tackle inter-layer and intra-layer redundancy: it introduces cross-layer-shared low-rank update matrices (A) and incorporates a runtime dynamic sparsity selection mechanism to sparsely update the (B) matrices on-demand. By unifying low-rank decomposition, parameter sharing, and dynamic sparse updates, EffiLoRA significantly reduces trainable parameters (up to 72% reduction) and FLOPs. Extensive experiments demonstrate that EffiLoRA consistently outperforms standard LoRA across diverse tasks—including commonsense reasoning, vision-language instruction tuning, and image generation—while preserving or improving model performance, enhancing resource efficiency, and boosting robustness. Moreover, EffiLoRA is readily extensible to multimodal architectures and diffusion models.

To address the excessive number of trainable parameters in large language model (LLM) fine-tuning, this paper proposes QR-LoRA, a low-rank adaptation method based on column-pivoted QR decomposition. Unlike standard LoRA or SVD-initialized variants, QR-LoRA efficiently constructs an orthogonal basis for pretrained weights via QR decomposition and represents adaptation updates as sparse linear combinations within this basis—optimizing only a small set of coefficients. This design jointly enhances computational efficiency and interpretability. QR-LoRA is the first to integrate a structured orthogonal basis into the low-rank adaptation framework. On the GLUE benchmark, it matches or surpasses full fine-tuning, standard LoRA, and SVD-LoRA in performance. With as few as 601 trainable parameters, it reduces parameter count by over 1,000× compared to full fine-tuning and by 77× relative to typical LoRA, achieving substantial gains in parameter efficiency.

LoRA’s training acceleration for large language models (LLMs) exhibits significant variability across architectures and training configurations, undermining its stability and portability. Through systematic empirical analysis across diverse architectures (e.g., LLaMA, Pythia, Qwen) and training scenarios (varying sequence lengths, batch sizes, and precisions), we identify three key bottlenecks limiting LoRA’s speedup: (i) memory-bandwidth-bound low-rank matrix memory accesses, (ii) computational redundancy in gradient aggregation during backward pass, and (iii) GPU memory fragmentation induced by adapter activation caching. To address these, we propose Bandwidth-Aware LoRA (BA-LoRA), a dynamic framework that jointly optimizes rank allocation and kernel fusion strategies based on runtime bandwidth constraints. Experiments demonstrate that BA-LoRA achieves 1.8–2.4× faster training than standard LoRA while preserving downstream task performance (within ±0.3% Acc/F1). Moreover, it reduces speedup variance by 67%, markedly improving cross-configuration robustness—enabling efficient, deployable fine-tuning under resource-constrained settings.

To address the substantial storage overhead and scalability challenges of LoRA modules in multi-task and personalized fine-tuning of large language models (LLMs), this paper proposes LoRA-XS—a radically lightweight low-rank adaptation method. Its core innovation lies in applying singular value decomposition (SVD) to pretrained weights, freezing the singular vectors, and introducing only a trainable $r imes r$ intermediate matrix—yielding an order-of-magnitude reduction in parameter count. This theory-driven architecture is the first to uncover structural redundancy in Transformer weight singular vectors while revealing their critical role in cross-task transfer. On a 7B model, LoRA-XS reduces parameters by over 100× compared to standard LoRA, yet matches or exceeds LoRA and VeRA performance on GLUE, GSM8K, MATH, and eight commonsense reasoning benchmarks. The method enables efficient deployment of million-scale personalized models.

To address the trade-off between efficiency and performance in low-rank adaptation (LoRA) for large language model fine-tuning, this paper proposes Continuous Low-rank Incremental Accumulation (CLIC). CLIC introduces, for the first time, an analytical column-wise scaling mechanism that dynamically approximates the optimal low-rank update direction without restarting optimization, thereby unifying high-rank expressivity with low computational overhead. The method integrates low-rank decomposition, gradient direction analysis, and analytically derived optimal scaling to enable parameter-efficient, cumulative updates. Experiments on a 12B-parameter model demonstrate that CLIC significantly outperforms existing LoRA variants across natural language understanding, commonsense reasoning, and mathematical reasoning tasks—achieving up to 35% faster convergence and final performance markedly closer to full fine-tuning.

Existing parameter-efficient fine-tuning (PEFT) methods—e.g., LoRA—perform low-rank updates across the full parameter space, introducing redundancy. We observe that pretrained models possess nontrivial null spaces, which naturally serve as effective subspaces for low-rank adaptation. To exploit this property, we propose Null-Space-constrained Low-Rank Adaptation (ZS-LoRA), the first method to strictly constrain incremental updates to the model’s null space. ZS-LoRA integrates parameter freezing and subspace optimization via singular value decomposition and orthogonal null-space projection. This design enhances effective rank and parameter efficiency without increasing trainable parameters. Empirically, ZS-LoRA achieves state-of-the-art performance on cross-modal retrieval and visual question answering tasks using significantly fewer tunable parameters. Our results demonstrate that null-space-constrained adaptation simultaneously improves both training efficiency and generalization capability, validating the null space as a principled and underutilized resource for PEFT.

This work addresses the inefficiency in existing Low-Rank Adaptation (LoRA) methods arising from the lack of coordinated optimization among rank allocation, scaling factors, and initialization. The authors propose TLoRA, a novel framework that, during early training, initializes the A matrix into a task-relevant subspace via singular value decomposition of the product between input activation covariances and pre-trained weights, then freezes A and trains only the B matrix. Simultaneously, TLoRA adaptively assigns per-layer ranks and scaling factors based on parameter sensitivity. This approach uniquely unifies initialization and resource allocation in LoRA under a task-aware paradigm, enabling highly efficient fine-tuning within a fixed parameter budget. Experiments demonstrate that TLoRA consistently achieves superior performance across diverse tasks—including natural language understanding, commonsense and mathematical reasoning, code generation, and dialogue—while substantially reducing the number of trainable parameters.

This work addresses the lack of theoretical understanding regarding the generalization gap between Low-Rank Adaptation (LoRA) and full fine-tuning. Within a simplified linear regression framework, the study systematically compares their generalization behaviors by integrating statistical learning theory with excess risk analysis. It establishes, for the first time, a theoretical guarantee that when the discrepancy between the pre-trained model and the downstream task exhibits a low-rank structure, LoRA achieves lower excess risk than full fine-tuning in both over-parameterized and under-parameterized regimes. Empirical experiments further corroborate this finding, demonstrating a non-intuitive improvement in test accuracy under low-rank constraints and thereby revealing the intrinsic mechanism underlying LoRA’s superior generalization capability.

This work addresses the parameter inefficiency of conventional LoRA-based continual learning, where each task is assigned a separate adapter despite significant low-rank redundancy across tasks. The authors propose LiteLoRA, a novel approach that leverages a plug-in gating mechanism and subspace overlap analysis to selectively reuse existing adapters or instantiate new ones in a dynamic, task-aware manner. By identifying and exploiting shared low-rank subspaces among tasks, LiteLoRA achieves competitive or superior performance compared to state-of-the-art methods on standard continual learning benchmarks, while reducing the number of active adapters by 20% to 70%, thereby substantially improving parameter efficiency.