Exploring Visual Prompting: Robustness Inheritance and Beyond

2025-06-07

AI Summary

This work presents the first systematic investigation into the transfer mechanisms of visual prompting (VP) under robust source models, revealing that VP can inherit source-model robustness but faces a pronounced trade-off between robustness and generalization. To address this, we propose Prompt Boundary Loosening (PBL), a lightweight, plug-and-play strategy that alleviates the trade-off by relaxing the optimization boundary of visual prompts. Grounded in visual prompt learning and robust transfer frameworks, PBL is rigorously evaluated across multiple downstream datasets along diverse dimensions. Experiments demonstrate that PBL not only preserves—often enhances—VP’s robustness but also significantly improves its cross-dataset generalization performance. Crucially, PBL achieves, for the first time, *simultaneous* improvement in both robustness inheritance and generalization capability, thereby validating its effectiveness and the broad applicability of our findings.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Adversarial Learning & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems

Abstract

Visual Prompting (VP), an efficient method for transfer learning, has shown its potential in vision tasks. However, previous works focus exclusively on VP from standard source models, it is still unknown how it performs under the scenario of a robust source model: Can the robustness of the source model be successfully inherited? Does VP also encounter the same trade-off between robustness and generalization ability as the source model during this process? If such a trade-off exists, is there a strategy specifically tailored to VP to mitigate this limitation? In this paper, we thoroughly explore these three questions for the first time and provide affirmative answers to them. To mitigate the trade-off faced by VP, we propose a strategy called Prompt Boundary Loosening (PBL). As a lightweight, plug-and-play strategy naturally compatible with VP, PBL effectively ensures the successful inheritance of robustness when the source model is a robust model, while significantly enhancing VP's generalization ability across various downstream datasets. Extensive experiments across various datasets show that our findings are universal and demonstrate the significant benefits of the proposed strategy.

Problem

• Investigates robustness inheritance in Visual Prompting from robust source models
• Examines trade-off between robustness and generalization in Visual Prompting
• Proposes Prompt Boundary Loosening to mitigate robustness-generalization trade-off

Innovation

• Visual Prompting inherits robust source model features
• Prompt Boundary Loosening mitigates robustness-generalization trade-off
• Lightweight plug-and-play strategy enhances downstream generalization