Rethinking Layer-Wise Information Allocation for Vision Foundation Model Adaptation

2026-07-24

AI Summary

Existing prompt tuning methods lack principled control over the allocation of information across layers in vision foundation models, leading to performance that is highly sensitive to prompt placement and limited generalization. This work proposes the Prompted Information Bottlenecks (PIB) framework, which introduces the information bottleneck principle into visual prompt tuning for the first time. By balancing information compression against task sufficiency layer by layer while keeping the backbone frozen, PIB effectively filters redundancy and preserves task-relevant information. The method provides a theoretical perspective on inter-layer information allocation, explaining the non-monotonic behavior of prompt capacity and significantly enhancing robustness and fine-grained recognition performance. With only 0.35% of parameters tuned, PIB achieves average accuracies of 92.1% on FGVC, 93.01% on HTA, and 77.33% on VTAB-1k across 34 datasets.

Abstract

Vision foundation models are increasingly reused as frozen backbones for downstream visual recognition, making parameter-efficient adaptation a central problem. Prompt-based adaptation, including Visual Prompt Tuning (VPT), provides a lightweight way to specialize these models, but its layer-wise behavior remains poorly understood: performance is sensitive to prompt depth, placement, and task distribution, and gains on standard in-domain benchmarks do not always translate into robust generalization. We argue that this limitation is not solely an optimization issue, but a layer-wise information allocation issue: existing prompt-based methods lack principled control over what prompt-conditioned representations should preserve, suppress, and propagate across depth. Inspired by the Information Bottleneck principle, we introduce Prompted Information Bottlenecks (PIB), a framework that regularizes layer-wise compression-sufficiency trade-offs and promotes a more coherent cross-layer information path. The key idea is that effective adaptation should be minimal yet sufficient, retaining task-relevant local evidence in earlier layers while progressively discarding nuisance factors and redundant details in deeper layers. Extensive experiments show that PIB achieves strong performance across 34 datasets, reaching 92.1% on FGVC, 93.01% on HTA, and 77.33% on VTAB-1k, while tuning only 0.35% parameters on average across the main settings. Beyond benchmark accuracy, PIB helps explain the non-monotonic behavior of prompt capacity scaling, reduces shortcut reliance, and improves robustness under distribution shift and fine-grained recognition settings. These results position PIB as both a practical method and an information-allocation perspective for adapting frozen vision foundation models. Our code is available at https://github.com/itsnotacie/MM-26-PIB

Problem

• prompt-based adaptation
• layer-wise information allocation
• vision foundation models
• generalization
• parameter-efficient adaptation

Innovation

• Prompted Information Bottlenecks
• layer-wise information allocation
• parameter-efficient adaptation
• vision foundation models
• information bottleneck