DenseLoRA: Dense Low-Rank Adaptation of Large Language Models

2025-05-27

AI Summary

To address parameter redundancy and low utilization efficiency in Low-Rank Adaptation (LoRA), this paper proposes DenseLoRA—a novel paradigm integrating representation refinement with dense low-rank adaptation. Its core innovation is the first introduction of a lightweight encoder-decoder architecture that jointly compresses and refines hidden states across all transformer layers; critically, it replaces LoRA’s conventional dual low-rank branches with a single dense low-rank matrix, enabling more efficient parameter usage. Evaluated on LLaMA3-8B, DenseLoRA achieves 83.8% accuracy using only 0.01% trainable parameters—substantially outperforming standard LoRA (0.70% parameters, 80.8% accuracy). This represents a breakthrough in both parameter efficiency and downstream task performance, demonstrating superior representational capacity and optimization effectiveness within constrained parameter budgets.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Low-rank adaptation (LoRA) has been developed as an efficient approach for adapting large language models (LLMs) by fine-tuning two low-rank matrices, thereby reducing the number of trainable parameters. However, prior research indicates that many of the weights in these matrices are redundant, leading to inefficiencies in parameter utilization. To address this limitation, we introduce Dense Low-Rank Adaptation (DenseLoRA), a novel approach that enhances parameter efficiency while achieving superior performance compared to LoRA. DenseLoRA builds upon the concept of representation fine-tuning, incorporating a single Encoder-Decoder to refine and compress hidden representations across all adaptation layers before applying adaptation. Instead of relying on two redundant low-rank matrices as in LoRA, DenseLoRA adapts LLMs through a dense low-rank matrix, improving parameter utilization and adaptation efficiency. We evaluate DenseLoRA on various benchmarks, showing that it achieves 83.8% accuracy with only 0.01% of trainable parameters, compared to LoRA's 80.8% accuracy with 0.70% of trainable parameters on LLaMA3-8B. Additionally, we conduct extensive experiments to systematically assess the impact of DenseLoRA's components on overall model performance. Code is available at https://github.com/mulin-ahu/DenseLoRA.

Problem

• Improves parameter efficiency in large language model adaptation
• Reduces redundancy in low-rank adaptation matrices
• Enhances adaptation performance with fewer trainable parameters

Innovation

• DenseLoRA enhances parameter efficiency with dense low-rank matrix
• Uses Encoder-Decoder to refine hidden representations before adaptation
• Achieves higher accuracy with fewer trainable parameters than LoRA