adapter design

This work addresses the challenges of multi-task adaptation and merging in parameter-efficient fine-tuning. We propose a modular architecture based on sparse adapters, integrating low-rank decomposition with structured weight sparsity to enable task-specific adapter training with minimal parameter updates—eliminating the need for downstream fine-tuning and supporting plug-and-play multi-task merging. Our key contribution is a streamlined, more efficient sparse training mechanism, both conceptually and empirically superior to LoRA and full-parameter fine-tuning. Experiments demonstrate the first successful joint merging of adapters across 20 diverse NLP tasks, yielding substantial gains in in-distribution performance. The approach consistently outperforms LoRA and full-model merging baselines in multi-task merging scenarios. However, cross-task generalization remains limited and warrants further investigation.

This work addresses the significant performance degradation often observed after merging Low-Rank Adaptation (LoRA) modules, a problem exacerbated by existing approaches that can only assess merge compatibility post-training, leading to wasted computational resources. The paper introduces MergeProbe, the first method capable of predicting post-merge performance retention early in training. MergeProbe leverages lightweight analysis of the alignment between low-rank updates and original gradients, as well as their perturbation of shared representations. It supports diverse composition strategies—including merging, reweighting, pruning, and routing—and consistently outperforms existing interference-aware merging techniques on the MERGE-PEFT benchmark across five domains: mathematics, code, science, instruction following, and safety. Notably, MergeProbe achieves superior average and worst-case performance retention while incurring substantially lower deployment overhead than full task routing.

Existing adapter-based fine-tuning methods, though parameter-efficient, suffer from substantial memory and computational overhead, prolonged training time, and imbalanced adapter contributions. This work is the first to empirically reveal the heterogeneous contribution of adapters within Transformer architectures. To address these issues, we propose a selective freezing mechanism: (i) dynamically assessing adapter importance via gradient sensitivity and task-specific contribution; (ii) implementing a staged, progressive freezing strategy; and (iii) incorporating implicit regularization to smooth the loss landscape and improve generalization. Our method maintains or even improves downstream task performance while significantly reducing memory consumption (−42.85%), FLOPs (−34.59%), and training time (−11.82%). The approach achieves superior efficiency without compromising robustness or accuracy, offering a principled and practical solution for resource-constrained adapter tuning.

Multi-task low-rank adaptation (LoRA) in large language models (LLMs) incurs substantial memory overhead due to task-specific adapter storage, while existing adapter merging methods suffer from significant performance degradation. Method: This paper proposes a tunable adapter merging framework grounded in matrix similarity, which quantifies the intrinsic similarity among LoRA weight matrices and formulates an optimization-driven merging strategy. Contribution/Results: Our approach enables continuous, controllable trade-offs between storage compression and task performance—overcoming the inflexibility of conventional fixed-compromise schemes. Empirical evaluation shows that, at just 52% of the original storage cost, the average performance drop is only 0.2–1.8%, substantially outperforming prior merging techniques. The method thus delivers both high efficiency and practical utility in resource-constrained deployment scenarios.

To address the dual challenge of multi-task generalization and computational efficiency in Automated Program Repair (APR) with Code Large Language Models (Code LLMs), this paper proposes a novel adapter-oriented continual fusion mechanism. Unlike conventional uniform or static-weighted adapter merging, our approach introduces task ordering and dynamic weighting into adapter composition—a first in Code LLM adaptation. Leveraging parameter-efficient fine-tuning, we empirically validate the method on CodeLlama: under optimal task sequences, continual fusion improves repair success rates by up to 12.7% over single-task adapters, significantly enhancing cross-task generalization while reducing inference overhead. Our core contribution lies in establishing the critical roles of task sequence and dynamic weight assignment in adapter fusion—thereby overcoming fundamental limitations of existing fusion paradigms.

This work addresses the challenge of efficiently constructing and managing massive numbers of persistent personalized models atop trillion-parameter foundation models. It proposes leveraging parameter-efficient fine-tuning (PEFT) as a lightweight and reliable personalization substrate, combining a shared large model with small, trainable adapters to encode user preferences, skills, and memory. The authors introduce MinT, an infrastructure that integrates adapter identity management, version control, provenance tracking, evaluation, and serving mechanisms, and define three scaling dimensions: Scale Up, Scale Down, and Scale Out. Experimental results demonstrate that, even under strong shared priors, compact adapters can stably capture personalized behaviors, offering a viable pathway toward large-scale deployment of millions of persistent personal models.

This work addresses the optimization instability and lack of theoretical capacity guidance associated with inserting adapters into frozen vision Transformer backbones during transfer learning. The authors propose the Zero-initialized Residual Low-rank Adapter, which introduces a low-rank bottleneck structure in each Transformer block, with the up-projection layer initialized to zero to ensure that fine-tuning starts identically to the pretrained model, thereby preventing early representation drift. For the first time, the adapter rank is theoretically modeled as a capacity budget tied to the feature shift of downstream tasks, revealing an “elbow”-shaped accuracy gain as rank increases. Experiments across nine datasets and three backbone scales show that the method improves top-1 accuracy by 14.9% on average over training only the classification head, using just 0.92% of the parameters required for full fine-tuning, and outperforms full fine-tuning in 10 out of 15 dataset-backbone combinations.

This work addresses a central challenge in parameter-efficient fine-tuning: identifying the optimal placement of adapters to achieve peak performance with minimal parameters. The authors propose PAGE, a metric based on initial gradient energy analysis, which reveals that adaptation effects are highly concentrated in the down-projection modules of shallow feed-forward networks. Leveraging this insight, they introduce DomLoRA—a method that deploys a single LoRA adapter exclusively in this dominant module. This study is the first to demonstrate the existence, architectural dependency, and task stability of such a dominant adaptation module, establishing a new paradigm for efficient fine-tuning. Experiments show that DomLoRA, using only ~0.7% of the parameters of standard LoRA, consistently outperforms it across diverse tasks—including instruction following, mathematical reasoning, code generation, and multi-turn dialogue—and further enhances the effectiveness of other LoRA variants.

Existing methods struggle to effectively merge multiple task-specific LoRA adapters into a single low-rank adapter without causing capability fragmentation or violating the low-rank structure. This work proposes a novel “Compress-then-Merge” (CtM) paradigm: it first constructs a shared r-dimensional subspace from the LoRA weights and orthogonally projects each adapter onto this subspace, then performs standard merging within the resulting r×r core coordinate space, followed by truncated SVD to strictly enforce the target rank constraint. Evaluated across multiple models and tasks, CtM significantly outperforms existing single-LoRA baselines and substantially narrows the performance gap with full-parameter merging, achieving for the first time an efficient and rank-preserving LoRA fusion.

Existing native 3D generative models struggle to achieve precise control over three-dimensional structures, particularly in fine-grained details such as joints and poses. To address this limitation, this work proposes SK-Adapter—a lightweight structural adapter that, for the first time, treats 3D skeletons as first-class control signals. By injecting learnable skeleton-encoded tokens into a frozen 3D generative backbone, SK-Adapter enables high-fidelity global and local structural manipulation. The approach leverages a cross-attention mechanism to integrate joint coordinates with topological structure and is trained on a newly curated large-scale dataset, Objaverse-TMS, comprising 24k text-mesh-skeleton triplets. Experiments demonstrate that SK-Adapter significantly outperforms existing methods in structural controllability while preserving geometric and textural quality, thereby overcoming the ambiguity inherent in conventional text- or image-based prompts.