Automatic Rank Allocation for Low-Rank Adaptation in Large Language Models via lp Regularization

2026-09-24

AI Summary

This study addresses the limitation of existing LoRA methods, whose rank allocation relies on manual heuristics lacking principled optimization objectives. We propose lp-LoRA, which introduces the lp sparsity-inducing technique from signal processing into low-rank adaptation for the first time. By integrating lp regularization with proximal operator optimization, our method establishes an automatic rank allocation mechanism that leverages implicit thresholding to prune redundant components, thereby enabling efficient fine-tuning. Extensive evaluations on natural language understanding and question answering tasks demonstrate that lp-LoRA achieves performance comparable to established LoRA baselines. Ultimately, this work provides a theoretically grounded and automated rank selection framework for parameter-efficient fine-tuning.

Abstract

Low-rank adaptation (LoRA) has become a popular parameter-efficient fine-tuning method for large language models. A key challenge in LoRA is how to determine the rank of each adaptation matrix, as rank directly controls its capacity and efficiency. Existing adaptive-rank methods typically allocate ranks according to manually designed importance scores, which are not directly derived from an optimization objective. In this work, we propose $\ell_p$-LoRA, a principled rank-allocation method based on $\ell_p$ regularization with $0<p<1$, which is a classical sparsity-inducing technique in signal processing and statistics. Specifically, we regularize the energy of each rank-one LoRA component, encouraging redundant components to vanish while preserving important ones. We derive the corresponding proximal subproblem and reduce the matrix optimization to a two-dimensional problem, leading to an implicit thresholding criterion for identifying redundant components. Experiments on natural language understanding and question-answering tasks demonstrate that the proposed method achieves competitive performance with existing LoRA baselines.

Problem

• Low-Rank Adaptation
• Rank Allocation
• Large Language Models
• Parameter-Efficient Fine-Tuning

Innovation

• Low-Rank Adaptation
• $\ell_p$ Regularization
• Automatic Rank Allocation
• Proximal Optimization
• Parameter-Efficient Fine-Tuning