FPT-Noise: Dynamic Scene-Aware Counterattack for Test-Time Adversarial Defense in Vision-Language Models

2025-10-22

AI Summary

Vision-language models (e.g., CLIP) exhibit strong zero-shot performance but are highly vulnerable to visual adversarial attacks; existing defenses typically require costly retraining. This paper proposes FPT-Noise, a test-time, fine-tuning-free defense that dynamically injects input-specific noise. It employs a scene-aware feature modulator and a feature-aware threshold to generate attack-adaptive, image-specific noise intensity, coupled with test-time transformation ensembling (TTE) for real-time robustness enhancement during inference. FPT-Noise significantly improves robustness against AutoAttack, raising average robust accuracy from 0.07% to 56.86%, while degrading clean accuracy by only 1.1%—outperforming prior test-time defenses. Its core innovations lie in a dynamic, perception-driven noise generation mechanism and a lightweight, deployment-efficient paradigm.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Privacy-enhancing technologies
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Vision-Language Models (VLMs), such as CLIP, have demonstrated remarkable zero-shot generalizability across diverse downstream tasks. However, recent studies have revealed that VLMs, including CLIP, are highly vulnerable to adversarial attacks, particularly on their visual modality. Traditional methods for improving adversarial robustness, such as adversarial training, involve extensive retraining and can be computationally expensive. In this paper, we propose a new Test-Time defense: Feature Perception Threshold Counterattack Noise (FPT-Noise), which enhances the adversarial robustness of CLIP without costly fine-tuning. Our core contributions are threefold: First, we introduce a Dynamic Feature Modulator that dynamically generate an image-specific and attack-adaptive noise intensity parameter. Second, We reanalyzed the image features of CLIP. When images are exposed to different levels of noise, clean images and adversarial images exhibit distinct rates of feature change. We established a feature perception threshold to distinguish clean images from attacked ones. Finally, we integrate a Scene-Aware Regulation guided by a stability threshold and leverage Test-Time Transformation Ensembling (TTE) to further mitigate the impact of residual noise and enhance robustness.Extensive experimentation has demonstrated that FPT-Noise significantly outperforms existing Test-Time defense methods, boosting average robust accuracy from 0.07% to 56.86% under AutoAttack while maintaining high performance on clean images (-1.1%). The code will be made public following the publication of the study. The code will be made public following the publication of the study.

Problem

• Defends vision-language models against adversarial attacks
• Enhances robustness without expensive fine-tuning process
• Distinguishes clean from adversarial images using dynamic thresholds

Innovation

• Dynamic Feature Modulator adapts noise to specific images and attacks
• Feature perception threshold distinguishes clean from adversarial images
• Scene-Aware Regulation and Test-Time Transformation Ensembling enhance robustness