OSoRA: Output-Dimension and Singular-Value Initialized Low-Rank Adaptation

2025-05-20

AI Summary

To address the high computational cost and low parameter efficiency in fine-tuning large language models (LLMs), this paper proposes SVLoRA, a novel low-rank adaptation method. SVLoRA initializes the low-rank decomposition via singular value decomposition (SVD) and uniquely jointly optimizes the output-dimensional vector and a learnable scaling factor—overcoming LoRA’s limitations of fixed rank and random initialization. Under backbone parameter freezing, SVLoRA achieves linear parameter growth while delivering substantial performance gains. Empirical evaluation across diverse benchmarks—including mathematical reasoning and commonsense reasoning—shows SVLoRA matches or surpasses LoRA and VeRA in accuracy, while reducing trainable parameters by 30–50%. The method thus achieves an improved trade-off between model expressivity and parameter efficiency, demonstrating both high accuracy and strong scalability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Fine-tuning Large Language Models (LLMs) has become increasingly challenging due to their massive scale and associated computational costs. Parameter-Efficient Fine-Tuning (PEFT) methodologies have been proposed as computational alternatives; however, their implementations still require significant resources. In this paper, we present OSoRA (Output-Dimension and Singular-Value Initialized Low-Rank Adaptation), a novel PEFT method for LLMs. OSoRA extends Low-Rank Adaptation (LoRA) by integrating Singular Value Decomposition (SVD) with learnable scaling vectors in a unified framework. It first performs an SVD of pre-trained weight matrices, then optimizes an output-dimension vector during training, while keeping the corresponding singular vector matrices frozen. OSoRA substantially reduces computational resource requirements by minimizing the number of trainable parameters during fine-tuning. Comprehensive evaluations across mathematical reasoning, common sense reasoning, and other benchmarks demonstrate that OSoRA achieves comparable or superior performance to state-of-the-art methods like LoRA and VeRA, while maintaining a linear parameter scaling even as the rank increases to higher dimensions. Our ablation studies further confirm that jointly training both the singular values and the output-dimension vector is critical for optimal performance.

Problem

• Reduces computational costs in fine-tuning large language models
• Improves parameter efficiency via SVD and learnable scaling vectors
• Maintains performance while minimizing trainable parameters

Innovation

• SVD-integrated LoRA with learnable scaling vectors
• Optimizes output-dimension vector, freezes singular vectors
• Reduces trainable parameters, maintains linear scaling