Block Circulant Adapter for Large Language Models

2025-05-01

AI Summary

To address the excessive parameter count and computational overhead in large language model (LLM) fine-tuning, this paper proposes a frequency-domain parameter-efficient fine-tuning method based on block-circulant matrices. It is the first to systematically integrate block-circulant matrix structure into LLM adapter design, leveraging one-dimensional discrete Fourier transform (DFT) to enable weight compression and low-rank updates in the frequency domain. By combining block-circulant decomposition with a gradient-stabilized update mechanism, the method jointly reduces both parameter count and FLOPs. Experiments demonstrate that, compared to VeRA and LoRA, it reduces parameters by 14× and 16×, respectively; relative to FourierFT, it cuts FLOPs by 32×; and it maintains competitive or superior performance across multiple downstream tasks. This approach overcomes efficiency bottlenecks of existing Fourier-domain fine-tuning methods and establishes a novel paradigm for computationally efficient LLM adaptation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Fine-tuning large language models (LLMs) is difficult due to their huge model size. Recent Fourier domain-based methods show potential for reducing fine-tuning costs. We propose a block circulant matrix-based fine-tuning method with a stable training heuristic to leverage the properties of circulant matrices and one-dimensional Fourier transforms to reduce storage and computation costs. Experiments show that our method uses $14 imes$ less number of parameters than VeRA, $16 imes$ smaller than LoRA and $32 imes$ less FLOPs than FourierFT, while maintaining close or better task performance. Our approach presents a promising way in frequency domain to fine-tune large models on downstream tasks.

Problem

• Reducing fine-tuning costs for large language models
• Leveraging circulant matrices to cut storage and computation
• Maintaining performance while using fewer parameters and FLOPs

Innovation

• Block circulant matrix-based fine-tuning method
• Leverages circulant matrices and Fourier transforms
• Reduces storage and computation costs significantly