adapt vit with lora

To address the challenge of efficient fine-tuning of vision large models under resource-constrained conditions, this paper proposes Serial LoRA—a novel low-rank adaptation method tailored for Vision Transformer (ViT) architectures. Its core innovation lies in serially embedding a shared low-rank matrix within attention modules, enabling structured parameter compression by exploiting intrinsic commonalities among adaptation parameters. Compared to standard LoRA, Serial LoRA reduces trainable parameters by 75% (requiring only 25% of the original count), significantly lowering memory footprint and GPU memory consumption, while maintaining comparable downstream task performance across multiple ViT backbones. The method integrates low-rank decomposition, attention module reconstruction, and cross-head/cross-layer parameter sharing—without introducing additional inference latency. Serial LoRA thus establishes a lightweight, general-purpose, and high-performance paradigm for parameter-efficient fine-tuning (PEFT), particularly suitable for edge-device deployment and large-scale applications.

To address the high computational resource demands of training large-scale Vision Transformers (ViTs), this paper proposes Dynamic Low-Rank Adaptation (Dynamic LoRA). Unlike static approaches, Dynamic LoRA initially performs full-parameter fine-tuning, then dynamically identifies layer-wise convergence by monitoring weight update magnitudes and adaptively transitions to hierarchical low-rank adaptation. Its key innovations include a module-level rank allocation strategy and a hyperparameter-driven phase-switching mechanism. Evaluated on ViT-Large, the method achieves lossless accuracy relative to full fine-tuning. Experiments demonstrate that Dynamic LoRA reduces trainable parameters by 90% (to 10% of full fine-tuning), triples training throughput, shortens per-epoch training time by 1.5×, and decreases GPU memory consumption by 20%. It consistently outperforms both static LoRA and full fine-tuning across efficiency and accuracy metrics.

To address the challenges of label scarcity and parameter inefficiency in cross-domain transfer of Vision Transformers (ViTs), this paper proposes an unsupervised, parameter-efficient extended pretraining paradigm. The method integrates Low-Rank Adaptation (LoRA) with partial unfreezing of only 1–2 Transformer blocks, while retaining established self-supervised objectives—either contrastive learning (DINOv2) or masked image modeling (MAE)—for unsupervised in-domain pretraining on target domains (e.g., satellite imagery). Crucially, no downstream supervision is required. Experimental results demonstrate substantial gains in linear probe performance: top-1 accuracy improves by up to 8% over full-parameter fine-tuning, achieving state-of-the-art performance with fewer than 10% trainable parameters. This work establishes a novel lightweight and adaptive paradigm for deploying large vision models across diverse domains without labeled data.

Catastrophic forgetting in vision transformers (ViTs) during sequential multi-task learning, exacerbated by domain shift in large-scale models. Method: We propose a dual low-rank adaptation framework: (i) orthogonal LoRA constrains parameter updates to the historical task subspace to preserve prior knowledge, while (ii) residual LoRA models new tasks exclusively in the orthogonal complement subspace; further enhanced by dynamic memory scheduling to balance stability and plasticity. The approach is built upon parameter-efficient fine-tuning (PEFT), integrating orthogonal subspace optimization with residual subspace decomposition. Results: Our method achieves state-of-the-art accuracy across multiple continual learning benchmarks—including Split-CIFAR100, Domain-Net, and CORe50—while reducing GPU memory consumption by up to 42% and accelerating inference by 1.8× compared to existing approaches. It consistently outperforms prior PEFT-based and replay-free continual learning methods in both classification accuracy and efficiency.

This work addresses the high computational complexity of gradient computation in Low-Rank Adaptation (LoRA) fine-tuning. Under the Strong Exponential Time Hypothesis (SETH), we establish the first computational phase-transition theory for LoRA gradient computation: we prove that a subquadratic approximation algorithm exists when the LoRA rank satisfies a norm-sharing upper-bound threshold; further, we design a chain-based low-rank approximation scheme enabling near-linear gradient updates. Methodologically, we propose a term-wise controlled analytical framework integrating fine-grained complexity analysis, hierarchical low-rank gradient approximation, and selective adaptation of attention weight subsets (Q/V/K). Our core contributions are: (1) the first characterization of the efficiency phase-transition threshold for LoRA gradient computation; (2) the identification of sufficient conditions for near-linear solvability; and (3) the first theory-driven acceleration pathway for low-rank adaptive methods.

This work addresses the limitation of fixed-rank constraints in parameter-efficient fine-tuning, which fail to accommodate the heterogeneous rank requirements across different layers of neural networks. The authors propose LR-LoRA, a novel approach that introduces a learnable rank mechanism within the LoRA framework, enabling differentiable and dynamic optimization of the rank for each adapter layer. This method reveals a systematic disparity in rank demands between attention and MLP layers in Transformers, thereby providing a more flexible and effective inductive bias. Experimental results demonstrate that LR-LoRA significantly outperforms existing parameter-efficient fine-tuning methods across multiple benchmarks for language understanding and commonsense reasoning, achieving state-of-the-art performance.

This study addresses the long-standing absence of theoretical guidance for selecting adapter ranks in LoRA fine-tuning, which complicates the trade-off between optimization conditioning and computational efficiency. Drawing on non-convex low-rank matrix sensing theory, this work proposes a data-dependent LoRA-RIP metric to characterize the optimization geometry of cross-entropy objectives. It further demonstrates that sufficient overparameterization eliminates spurious local minima, surpassing the classical 1/3 theoretical guarantee. Crucially, the analysis reveals that rank and data quality constitute coupled resources requiring joint consideration. Experiments across language and vision tasks validate these theoretical predictions, indicating that the proposed framework substantially enhances both the efficiency and reliability of parameter-efficient fine-tuning.

This work addresses the high computational cost of conventional fine-tuning for instance segmentation, which typically requires updating a large fraction (40–55%) of parameters in large pre-trained models. To improve parameter efficiency, the study explores parameter-efficient fine-tuning (PEFT) methods, introducing LoRA into deformable attention mechanisms for the first time and systematically evaluating the trade-offs between performance and efficiency based on the number and placement of adapters within the Transformer architecture. Experimental results demonstrate that by fine-tuning only 1–6% of the model parameters, the proposed approach matches or even surpasses the performance of full fine-tuning across four benchmark datasets. These findings validate the efficacy and feasibility of PEFT for instance segmentation and further reveal that its effectiveness is influenced by dataset complexity and model architecture.

Existing federated LoRA fine-tuning methods suffer from inconsistent aggregation, high communication overhead, manual hyperparameter tuning, and unstable convergence. This work proposes SpecTraL, the first approach to directly aggregate client-side LoRA modules in a low-rank latent space. By aligning subspaces via Householder orthogonal transformations and leveraging the spiked covariance model from random matrix theory, SpecTraL automatically discovers and efficiently aggregates layer-wise global ranks without dense reconstruction or auxiliary models. It introduces padding-aware initialization to avoid perturbing pre-trained weights and enables adaptive, hyperparameter-free rank selection. Evaluated on DomainNet and NICO++ with ViT-B/16 and ViT-L/16, SpecTraL significantly improves the trade-off between accuracy and communication efficiency, reduces server-side computation, and eliminates the need for rank hyperparameter search.

Deploying billion-parameter vision–language–action (VLA) models on industrial robots is hindered by the embodiment gap and the high computational cost of full fine-tuning (FFT). This work systematically evaluates low-rank adaptation (LoRA) for fine-tuning the π₀ flow-matching VLA model, conducting four precision assembly tasks on a UR5e robotic arm while exploring varying ranks, parameter allocation schemes, and module-freezing strategies. The experiments demonstrate that LoRA with rank r=32, uniformly applied across the vision encoder, language model, and action expert modules, achieves performance on par with FFT, whereas freezing any core backbone component significantly degrades task success. This approach reduces peak static GPU memory from 36.2 GiB to 10.8 GiB, revealing for the first time that effective embodied adaptation requires concurrent preservation of both semantic and visual plasticity, thereby establishing a practical paradigm for efficient deployment.