FRoD: Full-Rank Efficient Fine-Tuning with Rotational Degrees for Fast Convergence

2025-12-29

AI Summary

Existing parameter-efficient fine-tuning (PEFT) methods—such as LoRA—are constrained by low-rank approximations, leading to slow convergence, limited expressivity, and difficulty in capturing complex downstream task patterns. To address this, we propose HR-LoRA, a novel PEFT framework integrating hierarchical joint decomposition with rotational degree-of-freedom modeling. Its core innovation lies in a globally shared low-dimensional basis matrix coupled with task-adaptive sparse scaling perturbations, enabling near-full-rank parameter updates while maintaining an extremely low number of trainable parameters. This design bridges the expressivity gap between PEFT and full fine-tuning. Evaluated across 20 cross-modal benchmarks, HR-LoRA achieves performance on par with full fine-tuning using only 1.72% trainable parameters, while substantially accelerating convergence and improving out-of-distribution robustness.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Learning to Search
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Parameter-efficient fine-tuning (PEFT) methods have emerged as a practical solution for adapting large foundation models to downstream tasks, reducing computational and memory costs by updating only a small subset of parameters. Among them, approaches like LoRA aim to strike a balance between efficiency and expressiveness, but often suffer from slow convergence and limited adaptation capacity due to their inherent low-rank constraints. This trade-off hampers the ability of PEFT methods to capture complex patterns needed for diverse tasks. To address these challenges, we propose FRoD, a novel fine-tuning method that combines hierarchical joint decomposition with rotational degrees of freedom. By extracting a globally shared basis across layers and injecting sparse, learnable perturbations into scaling factors for flexible full-rank updates, FRoD enhances expressiveness and efficiency, leading to faster and more robust convergence. On 20 benchmarks spanning vision, reasoning, and language understanding, FRoD matches full model fine-tuning in accuracy, while using only 1.72% of trainable parameters under identical training budgets.

Problem

• Addresses slow convergence in parameter-efficient fine-tuning methods
• Overcomes limited adaptation capacity due to low-rank constraints
• Enhances expressiveness and efficiency for diverse tasks

Innovation

• Hierarchical joint decomposition with rotational degrees
• Sparse learnable perturbations in scaling factors
• Full-rank updates for faster robust convergence