AdapterTune: Zero-Initialized Low-Rank Adapters for Frozen Vision Transformers

2026-03-15

AI Summary

This work addresses the optimization instability and lack of theoretical capacity guidance associated with inserting adapters into frozen vision Transformer backbones during transfer learning. The authors propose the Zero-initialized Residual Low-rank Adapter, which introduces a low-rank bottleneck structure in each Transformer block, with the up-projection layer initialized to zero to ensure that fine-tuning starts identically to the pretrained model, thereby preventing early representation drift. For the first time, the adapter rank is theoretically modeled as a capacity budget tied to the feature shift of downstream tasks, revealing an “elbow”-shaped accuracy gain as rank increases. Experiments across nine datasets and three backbone scales show that the method improves top-1 accuracy by 14.9% on average over training only the classification head, using just 0.92% of the parameters required for full fine-tuning, and outperforms full fine-tuning in 10 out of 15 dataset-backbone combinations.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Large Vision Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Frozen-backbone transfer with Vision Transformers faces two under-addressed issues: optimization instability when adapters are naively inserted into a fixed feature extractor, and the absence of principled guidance for setting adapter capacity. We introduce AdapterTune, which augments each transformer block with a residual low-rank bottleneck whose up-projection is zero-initialized, guaranteeing that the adapted network starts exactly at the pretrained function and eliminates early-epoch representation drift. On the analytical side, we formalize adapter rank as a capacity budget for approximating downstream task shifts in feature space. The resulting excess-risk decomposition predicts monotonic but diminishing accuracy gains with increasing rank, an ``elbow'' behavior we confirm through controlled sweeps. We evaluate on 9 datasets and 3 backbone scales with multi-seed reporting throughout. On a core 5 dataset transfer suite, AdapterTune improves top-1 accuracy over head-only transfer by +14.9 points on average while training only 0.92 of the parameters required by full fine-tuning, and outperforms full fine-tuning on 10 of 15 dataset-backbone pairs. Across the full benchmark, AdapterTune improves over head-only transfer on every dataset-backbone pair tested. Ablations on rank, placement, and initialization isolate each design choice. The code is available at: https://github.com/salimkhazem/adaptertune

Problem

• frozen-backbone transfer
• optimization instability
• adapter capacity
• Vision Transformers
• low-rank adapters

Innovation

• zero-initialized adapters
• low-rank adaptation
• frozen Vision Transformers
• capacity budget
• representation drift