MoS: Unleashing Parameter Efficiency of Low-Rank Adaptation with Mixture of Shards

2024-10-01

arXiv.org

AI Summary

To address the high GPU memory overhead and low parameter efficiency in multi-task customized large language model serving, this paper proposes Mixture-of-Shards (MoS), an efficient fine-tuning method. Built upon the LoRA framework, MoS innovatively integrates Mixture-of-Experts (MoE)-style routing with four zero-cost differentiation mechanisms: subset selection, decoupled alignment, vector sharding, and shard privatization—enabling both cross-layer and intra-layer parameter sharing while preserving model performance and enhancing task discrimination. Experiments demonstrate that MoS achieves approximately 8× parameter compression over standard LoRA, significantly outperforming existing shared-parameter fine-tuning approaches. Ablation studies confirm the individual efficacy and synergistic benefits of each differentiation component.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Large language models for search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

The rapid scaling of large language models necessitates more lightweight finetuning methods to reduce the explosive GPU memory overhead when numerous customized models are served simultaneously. Targeting more parameter-efficient low-rank adaptation (LoRA), parameter sharing presents a promising solution. Empirically, our research into high-level sharing principles highlights the indispensable role of differentiation in reversing the detrimental effects of pure sharing. Guided by this finding, we propose Mixture of Shards (MoS), incorporating both inter-layer and intra-layer sharing schemes, and integrating four nearly cost-free differentiation strategies, namely subset selection, pair dissociation, vector sharding, and shard privatization. Briefly, it selects a designated number of shards from global pools with a Mixture-of-Experts (MoE)-like routing mechanism before sequentially concatenating them to low-rank matrices. Hence, it retains all the advantages of LoRA while offering enhanced parameter efficiency, and effectively circumvents the drawbacks of peer parameter-sharing methods. Our empirical experiments demonstrate approximately 8x parameter savings in a standard LoRA setting. The ablation study confirms the significance of each component. Our insights into parameter sharing and MoS method may illuminate future developments of more parameter-efficient finetuning methods. The code is officially available at https://github.com/Forence1999/MoS.

Problem

• Enhance parameter efficiency in finetuning
• Reduce GPU memory overhead
• Implement effective parameter sharing strategies

Innovation

• Mixture of Shards (MoS)
• Low-Rank Adaptation (LoRA)
• Parameter Efficiency