IPA: An Information-Preserving Input Projection Framework for Efficient Foundation Model Adaptation

2025-09-04

AI Summary

Existing parameter-efficient fine-tuning (PEFT) methods—such as LoRA—suffer from input feature information loss under random initialization of projection matrices, and exhibit limited projection updates during training, forming a critical performance bottleneck. This paper proposes Information-Preserving Input Projection (IPA), a novel PEFT framework that introduces the first data-aware feature preservation mechanism. IPA precomputes the projection matrix via approximate Principal Component Analysis (PCA) on task-specific data, enabling effective information compression while keeping the projection fixed during fine-tuning. Compared to standard LoRA, IPA achieves comparable performance using only 50% of the trainable parameters. On diverse language and vision multitask benchmarks, IPA consistently improves commonsense reasoning accuracy by +1.5 percentage points and VTAB-1k score by +2.3 points. It significantly reduces the number of tunable parameters while enhancing generalization across domains.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: Safety and Robustness
• Computer Vision: Large Vision Models

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, reduce adaptation cost by injecting low-rank updates into pretrained weights. However, LoRA's down-projection is randomly initialized and data-agnostic, discarding potentially useful information. Prior analyses show that this projection changes little during training, while the up-projection carries most of the adaptation, making the random input compression a performance bottleneck. We propose IPA, a feature-aware projection framework that explicitly preserves information in the reduced hidden space. In the linear case, we instantiate IPA with algorithms approximating top principal components, enabling efficient projector pretraining with negligible inference overhead. Across language and vision benchmarks, IPA consistently improves over LoRA and DoRA, achieving on average 1.5 points higher accuracy on commonsense reasoning and 2.3 points on VTAB-1k, while matching full LoRA performance with roughly half the trainable parameters when the projection is frozen.

Problem

• Preserves information in low-rank adaptation
• Addresses random initialization in LoRA projections
• Improves efficiency and accuracy in model fine-tuning

Innovation

• Feature-aware projection preserving information in hidden space
• Algorithms approximating top principal components for efficiency
• Pretrains projectors with negligible inference overhead