FREE-Merging: Fourier Transform for Model Merging with Lightweight Experts

2024-11-25

arXiv.org

AI Summary

To address the challenge of task interference in model merging—where performance degradation and deployment overhead hinder simultaneous optimization—this work identifies, for the first time, that interference manifests prominently in the frequency domain, whereas existing methods operate solely in the spatial domain and thus suffer from limited efficacy. We propose a lightweight, Fourier-transform-based expert-augmented fusion framework: (1) a novel frequency-domain filtering mechanism to suppress harmful fine-tuning signals; (2) dynamically activated low-rank expert modules that compensate for information loss at zero training cost; and (3) a unified cross-modal fusion architecture. Evaluated across CV, NLP, and multimodal benchmarks, our method consistently outperforms state-of-the-art approaches, achieving a 37% inference speedup, 52% reduction in parameter storage, and preserving ≥98.6% single-task accuracy.

Technology Category

Machine Learning: Mixture of Experts (MoE)Computer Vision: Multi-modal VisionIntelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsSearch and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAGUser Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

In the current era of rapid expansion in model scale, there is an increasing availability of open-source model weights for various tasks. However, the capabilities of a single fine-tuned model often fall short of meeting diverse deployment needs. Model merging has thus emerged as a widely focused method for efficiently building a single model tailored for multiple tasks combined from existing models. Nevertheless, existing model merging methods face challenging trade-offs between performance and deployment costs, primarily due to task conflicts within the merged network. Our analysis of neural networks reveals that some task-specific information introduced by fine-tuning minimally enhances performance but heavily impacts generalization, leading to task conflicts. To mitigate the impact of this information, we propose FR-Merging, an innovative method that leverages frequency domain information to efficiently filter harmful specialized information, thereby minimizing the impact of task conflicts on the backbone with minimal cost. Since performance loss is inevitable with cost-free merging methods, we introduce a lightweight task-specific expert that can be dynamically integrated during inference to compensate for information loss. This framework, FREE-Merging (FR-Merging with lightweight experts), strikes a balanced trade-off between training cost, inference speed, storage requirements, and performance. We demonstrate the effectiveness of both FR-Merging and FREE-Merging on multiple tasks across CV, NLP, and Multi-Modal domains and show that they can be flexibly adapted to meet specific needs.

Problem

• Addresses task interference in model merging via frequency domain analysis
• Proposes FR-Merging to filter harmful frequency interference efficiently
• Introduces FREE-Merging framework to balance cost, latency, and performance

Innovation

• Fourier Transform filters frequency domain interference
• Lightweight expert module compensates information loss
• Balances cost, latency, storage, and performance