AROMA: Autonomous Rank-one Matrix Adaptation

2025-04-06

AI Summary

To address the performance limitations of static or heuristic rank allocation in parameter-efficient fine-tuning (PEFT) of large language models (LLMs), this paper proposes AROMA: an automatic, layer-adaptive framework for incremental rank-one matrix construction. Its core innovation is a novel dual-loop optimization mechanism—inner loops sequentially extract orthogonal rank-one subspaces, while outer loops autonomously determine the optimal rank per layer. To ensure subspace independence, AROMA resets optimizer states between iterations and replaces conventional rank reduction with structured sparsification via parameter freezing and progressive zeroing. Experiments demonstrate that AROMA significantly reduces trainable parameters—by several-fold compared to LoRA and AdaLoRA—while achieving superior performance on natural language understanding and commonsense reasoning benchmarks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

As large language models continue to grow in size, parameter-efficient fine-tuning has become increasingly crucial. While low-rank adaptation (LoRA) offers a solution through low-rank updates, its static rank allocation may yield suboptimal results. Adaptive low-rank adaptation (AdaLoRA) improves this with dynamic allocation but remains sensitive to initial and target rank configurations. We introduce AROMA, a framework that automatically constructs layer-specific updates by iteratively building up rank-one components with very few trainable parameters that gradually diminish to zero. Unlike existing methods that employ rank reduction mechanisms, AROMA introduces a dual-loop architecture for rank growth. The inner loop extracts information from each rank-one subspace, while the outer loop determines the number of rank-one subspaces, i.e., the optimal rank. We reset optimizer states to maintain subspace independence. AROMA significantly reduces parameters compared to LoRA and AdaLoRA while achieving superior performance on natural language understanding and commonsense reasoning tasks, offering new insights into adaptive parameter-efficient fine-tuning. The code is available at href{https://github.com/ShuDun23/AROMA}{AROMA}.

Problem

• Dynamic rank allocation for efficient fine-tuning
• Reducing trainable parameters in large language models
• Improving performance on NLP and reasoning tasks

Innovation

• Iteratively builds rank-one components
• Uses dual-loop architecture for rank growth
• Resets optimizer states for subspace independence