Adapting Vision Foundation Models with Cascaded Semantics

2026-08-05

AI Summary

Existing visual prompt tuning methods rely on random initialization and overlook semantic priors, thereby limiting the adaptability of pretrained Vision Transformers (ViTs) to downstream tasks. This work proposes a cascaded prompt tuning framework that, for the first time, integrates handcrafted input-space priors—such as color, texture, and shape—with instance-aware self-attention semantics in feature space. By synergistically combining these complementary dual-level priors, the framework effectively guides prompt learning. Remarkably, the method achieves state-of-the-art performance across 34 image classification benchmarks while tuning only 0.74% of the ViT parameters, demonstrating both high efficiency and strong adaptation capability for downstream tasks.

Abstract

Prompt tuning, a leading parameter-efficient adaptation paradigm in NLP, has recently been extended to computer vision. Visual prompt tuning (VPT) adapts pre-trained vision transformers (ViTs) by updating a small set of additional prompt parameters. However, existing visual prompts are randomly initialized and do not exploit prior knowledge, such as instructions in NLP. We address this gap by injecting two complementary semantic priors into VPT. Fundamental image priors, including color, texture, and shape, are extracted with classical hand-crafted operators and injected into the input space, while self-attention maps provide instance-aware semantics in the feature space. We further propose a cascaded scheme that integrates both priors throughout ViT adaptation. Experiments on 34 challenging image classification datasets demonstrate superior downstream adaptation while tuning only 0.74% of ViT parameters. Project page: https://xixiaouab.github.io/Cascaded-Semantics/.

Problem

• visual prompt tuning
• vision foundation models
• semantic priors
• parameter-efficient adaptation
• vision transformers

Innovation

• visual prompt tuning
• semantic priors
• cascaded adaptation
• vision transformers
• parameter-efficient learning