Adaptive Utilization of Low-Rank Adaptation via Conditioned Gating

2026-10-05

AI Summary

This study addresses the limitation of standard LoRA, where shared low-rank update matrices restrict a model’s capacity to adapt differentially across sequence tokens. To overcome this, we propose U-LoRA, which introduces a conditional gating mechanism for token-level adaptive selection of low-rank subspaces, stabilized via bias-corrected exponential moving average (EMA). Rather than directly expanding subspace dimensions, our core innovation lies in optimizing the utilization of existing subspaces through input-conditioned strategies, thereby enhancing adaptation consistency in parameter-efficient fine-tuning. Empirical evaluations on mathematical reasoning and natural language understanding benchmarks demonstrate that U-LoRA significantly outperforms strong LoRA baselines under an equivalent parameter budget.

Abstract

Low-Rank Adaptation (LoRA) achieves parameter-efficient fine-tuning by constraining model updates to a low-rank subspace and has been widely used in practice. However, LoRA typically employs a shared low-rank update across tokens, which limits its ability to fully exploit the adaptation subspace for tokens from different sequences. To address this issue, we propose an adaptive utilization of Low-Rank Adaptation (U-LoRA), which employs conditioned gating to explicitly learn effective token-level utilization of the limited low-rank adaptation subspace. Specifically, U-LoRA generates utilization coefficients along low-rank directions for each token and jointly coordinates and constrains them using sequence-level contextual information, thereby inducing more consistent adaptive patterns within a sentence. To further enhance training stability, we introduce a bias-corrected exponential moving average (EMA) historical prior that calibrates utilization signals across optimization steps, suppressing noise caused by batch-to-batch fluctuations. The effectiveness of our method arises from a better utilization of the existing low-rank subspace via input-conditioned strategies, rather than from expanding the subspace. Experiments on mathematical reasoning and natural language understanding benchmarks demonstrate that U-LoRA achieves competitive performance under comparable parameter budgets when with strong LoRA baselines and recent variants.

Problem

• Low-Rank Adaptation
• Parameter-efficient fine-tuning
• Token-level adaptation
• Shared low-rank update

Innovation

• Low-Rank Adaptation
• Conditioned Gating
• Token-level Utilization
• Exponential Moving Average
• Parameter-Efficient Fine-Tuning