Structure-Learnable Adapter Fine-Tuning for Parameter-Efficient Large Language Models

2025-09-03

AI Summary

To address parameter redundancy, structural rigidity, and insufficient task adaptability in large language model (LLM) fine-tuning, this paper proposes a learnable-structure adapter tuning method. Under frozen backbone parameters, it jointly optimizes adapter insertion positions, activation paths, and module compositions via differentiable gating and structural sparsity control, dynamically constructing task-specific, efficient substructures. Its key innovation lies in formulating architecture search as a differentiable optimization problem, integrating sensitivity analysis to quantify the impact of sparsity, noise, and data perturbations. Experiments across multiple natural language understanding benchmarks demonstrate that the method significantly outperforms mainstream parameter-efficient fine-tuning approaches—achieving higher accuracy, superior parameter compression ratios, and enhanced robustness against input noise.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Structured Learning
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

This paper addresses the issues of parameter redundancy, rigid structure, and limited task adaptability in the fine-tuning of large language models. It proposes an adapter-based fine-tuning method built on a structure-learnable mechanism. By introducing differentiable gating functions and structural sparsity control variables, the method enables automatic optimization of adapter insertion points, activation paths, and module combinations. This allows the model to adjust its structure flexibly in multi-task settings to match different task characteristics. With the backbone parameters kept frozen, the method uses a structure search mechanism to guide the dynamic construction of task-specific efficient substructures during training. This significantly improves parameter utilization and representational capacity. In addition, the paper designs a set of sensitivity analysis experiments to systematically evaluate the effects of sparsity weight, noise injection ratio, and data perturbation on model performance. These experiments verify the stability and robustness of the proposed method across various multi-task natural language understanding tasks. The experimental results show that the proposed method outperforms mainstream parameter-efficient tuning techniques on multiple tasks. It achieves a better balance among accuracy, compression rate, and robustness to noise and perturbation.

Problem

• Reduces parameter redundancy in fine-tuning large models
• Enhances task adaptability through learnable adapter structures
• Improves robustness against noise and data perturbations

Innovation

• Structure-learnable adapter fine-tuning mechanism
• Differentiable gating functions with sparsity control
• Dynamic construction of task-specific efficient substructures