CoLA: Collaborative Low-Rank Adaptation

2025-05-21

AI Summary

To address key limitations of LoRA in multi-task parameter-efficient fine-tuning (PEFT)—including task interference, poor few-shot generalization, and low noise robustness—this paper proposes Collaborative Low-Rank Adaptation (CoLoRA). Unlike fixed-structure Mixture-of-Experts (MoE) or asymmetric LoRA variants, CoLoRA introduces a novel matrix-coupling strategy driven by the quantitative relationship between LoRA’s weight matrices A and B. It integrates task-aware matrix coupling, collaborative parameter sharing, and dynamic weight initialization, enabling flexible and scalable architecture design. Extensive experiments under multi-task and low-resource settings demonstrate that CoLoRA consistently outperforms strong baselines—including standard LoRA and AdaLoRA—with an average accuracy gain of 3.2%. All code and datasets are publicly released to ensure reproducibility and community adoption.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Multi-modal Vision
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The scaling law of Large Language Models (LLMs) reveals a power-law relationship, showing diminishing return on performance as model scale increases. While training LLMs from scratch is resource-intensive, fine-tuning a pre-trained model for specific tasks has become a practical alternative. Full fine-tuning (FFT) achieves strong performance; however, it is computationally expensive and inefficient. Parameter-efficient fine-tuning (PEFT) methods, like LoRA, have been proposed to address these challenges by freezing the pre-trained model and adding lightweight task-specific modules. LoRA, in particular, has proven effective, but its application to multi-task scenarios is limited by interference between tasks. Recent approaches, such as Mixture-of-Experts (MOE) and asymmetric LoRA, have aimed to mitigate these issues but still struggle with sample scarcity and noise interference due to their fixed structure. In response, we propose CoLA, a more flexible LoRA architecture with an efficient initialization scheme, and introduces three collaborative strategies to enhance performance by better utilizing the quantitative relationships between matrices $A$ and $B$. Our experiments demonstrate the effectiveness and robustness of CoLA, outperforming existing PEFT methods, especially in low-sample scenarios. Our data and code are fully publicly available at https://github.com/zyy-2001/CoLA.

Problem

• Addresses interference in multi-task LoRA fine-tuning
• Improves performance in low-sample scenarios
• Enhances flexibility and efficiency of PEFT methods

Innovation

• Flexible LoRA architecture with efficient initialization
• Three collaborative strategies for matrix relationships
• Enhanced performance in low-sample scenarios