MaCP: Minimal yet Mighty Adaptation via Hierarchical Cosine Projection

2025-05-29

AI Summary

To address the challenge of balancing accuracy against parameter and memory overhead in efficient large-model fine-tuning, this paper proposes Hierarchical Cosine Projection Adaptation (HCPA). Its core innovation lies in the first application of low-rank weight updates mapped into the Discrete Cosine Transform (DCT) frequency domain, where hierarchical spectral decomposition dynamically identifies energy-concentrated, decorrelated critical frequency components—enabling joint optimization of frequency-domain sparsification and structured low-rank adaptation. HCPA seamlessly integrates with the LoRA framework. Extensive experiments across diverse tasks—including natural language understanding (NLU), natural language generation (NLG), summarization, image classification, and video understanding—demonstrate that HCPA consistently outperforms existing lightweight fine-tuning methods: average accuracy improves by 1.2–2.8%, computational complexity decreases by over 35%, and GPU memory consumption is reduced by more than 40%.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

We present a new adaptation method MaCP, Minimal yet Mighty adaptive Cosine Projection, that achieves exceptional performance while requiring minimal parameters and memory for fine-tuning large foundation models. Its general idea is to exploit the superior energy compaction and decorrelation properties of cosine projection to improve both model efficiency and accuracy. Specifically, it projects the weight change from the low-rank adaptation into the discrete cosine space. Then, the weight change is partitioned over different levels of the discrete cosine spectrum, and each partition's most critical frequency components are selected. Extensive experiments demonstrate the effectiveness of MaCP across a wide range of single-modality tasks, including natural language understanding, natural language generation, text summarization, as well as multi-modality tasks such as image classification and video understanding. MaCP consistently delivers superior accuracy, significantly reduced computational complexity, and lower memory requirements compared to existing alternatives.

Problem

• Efficient fine-tuning of large foundation models
• Improving model accuracy with minimal parameters
• Reducing computational complexity and memory usage

Innovation

• Hierarchical cosine projection for adaptation
• Low-rank weight change in cosine space
• Critical frequency components selection