Parameter-Efficient Fine-Tuning for Foundation Models

2025-01-23

AI Summary

Parameter-efficient fine-tuning (PEFT) methods for large foundation models suffer from unclear cross-modal adaptation mechanisms and a lack of systematic classification frameworks. Method: This paper introduces the first unified PEFT taxonomy spanning language, vision, and multimodal foundation models, systematically analyzing the adaptation principles and architectural distinctions of mainstream techniques—including LoRA, Adapter, Prompt Tuning, and IA³—within the Transformer architecture. Through comparative analysis, it identifies evolutionary trends and critical research gaps. Contribution/Results: We publicly release an authoritative, curated PEFT literature repository on GitHub, featuring a structured knowledge graph and practical implementation guidelines. This resource significantly lowers the barrier to task-specific customization of large models and establishes foundational support for advancing PEFT theory and enabling robust cross-modal applications.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Natural Language Processing: Language Grounding & Multi-modal NLPComputer Vision: Large Vision Models

Application Category

Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphsSearch and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAGWeb Mining and Content Analysis: Large pretrained models with web data

Abstract

This survey delves into the realm of Parameter-Efficient Fine-Tuning (PEFT) within the context of Foundation Models (FMs). PEFT, a cost-effective fine-tuning technique, minimizes parameters and computational complexity while striving for optimal downstream task performance. FMs, like ChatGPT, DALL-E, and LLaVA specialize in language understanding, generative tasks, and multimodal tasks, trained on diverse datasets spanning text, images, and videos. The diversity of FMs guides various adaptation strategies for PEFT. Therefore, this survey aims to provide a comprehensive overview of PEFT techniques applied to diverse FMs and address critical gaps in understanding the techniques, trends, and applications. We start by providing a detailed development of FMs and PEFT. Subsequently, we systematically review the key categories and core mechanisms of PEFT across diverse FMs to offer a comprehensive understanding of trends. We also explore the most recent applications across various FMs to demonstrate the versatility of PEFT, shedding light on the integration of systematic PEFT methods with a range of FMs. Furthermore, we identify potential research and development directions for improving PEFTs in the future. This survey provides a valuable resource for both newcomers and experts seeking to understand and use the power of PEFT across FMs. All reviewed papers are listed at url{https://github.com/THUDM/Awesome-Parameter-Efficient-Fine-Tuning-for-Foundation-Models}.

Problem

• Parameter-Efficient Fine-Tuning
• Large Models
• Cross-Task Performance

Innovation

• Parameter Efficient Fine-Tuning
• multi-modal data learning
• large foundation models