ReACT-CLIP: Response-Aware Test-Time Defense for Vision--Language Models

2026-08-02

AI Summary

Existing test-time defense methods that require no retraining suffer from significantly degraded robustness under unknown attack strengths due to their use of fixed correction intensities. This work proposes a response-aware adaptive defense mechanism that, for the first time, introduces a joint dynamic regulation strategy based on visual feature drift and prediction instability. Specifically, it constructs stable feature anchors by measuring relative feature drift between high- and low-noise probes to map an appropriate Gaussian noise scale, and employs Jensen–Shannon divergence to quantify prediction instability, thereby dynamically determining—on a per-sample basis—whether to intervene and at what correction intensity. Without modifying or retraining the model, the method substantially improves robustness across diverse attack types and strengths on twelve downstream tasks, including ImageNet and its distribution-shifted variants, while incurring negligible loss in clean accuracy.

Abstract

Training-free test-time defenses offer a practical way to improve the adversarial robustness of CLIP-style vision--language models without modifying the pretrained model. However, their correction strength is typically fixed for a narrow range of attack budgets, even though the attack budget is unknown at inference and the required correction varies across samples. We show that this mismatch causes existing defenses to degrade sharply as attacks strengthen. We introduce ReACT-CLIP, a response-conditioned test-time defense that separately determines how strongly each input should be corrected and whether defensive intervention is necessary. Our key observation is that the relative increase in CLIP visual-feature drift between low- and high-noise probes provides a graded, sample-specific proxy for correction demand. ReACT-CLIP maps this relative cross-noise drift to the Gaussian noise scale used to construct a stable, noise-averaged feature anchor, enabling the corrective reach to adapt to each input. To determine whether intervention is necessary, we further observe that clean inputs retain stable class-probability distributions under weak spatial augmentations, whereas adversarial inputs exhibit greater variation. ReACT-CLIP quantifies this variation using a prediction-instability score computed by Jensen--Shannon divergence and combines it with relative cross-noise drift to form the defensive intervention score. ReACT-CLIP requires no model or prompt training, and its correction-strength mapping is calibrated once and fixed across datasets and attack budgets. Across 12 downstream datasets, as well as ImageNet and its distribution-shifted variants, ReACT-CLIP delivers substantial robustness gains across diverse attack types and strengths while largely preserving clean accuracy.

Problem

• adversarial robustness
• vision-language models
• test-time defense
• attack budget
• correction strength

Innovation

• test-time defense
• adversarial robustness
• vision-language models
• response-aware adaptation
• feature drift