Model Confidence Under Answer-Preserving Attacks: An Informativeness-Manipulability Frontier

2026-08-06

AI Summary

Current vision-language systems rely on confidence scores for answer filtering, yet their robustness under adversarial attacks that strictly preserve the exact byte sequence of generated answers remains unclear. This work proposes a white-box image perturbation method that systematically manipulates model confidence while keeping the generated answer completely unchanged. We demonstrate for the first time that confidence can be significantly suppressed even under this stringent constraint, and we introduce a verifiable robustness criterion based on uniform-magnitude certificates along with an analysis of its failure mechanisms. Experiments show that the proposed attack is effective across all tested models and benchmarks, reducing confidence below the accuracy baseline; in gating simulations, it leads to an erroneous answer acceptance rate as high as 84.8%, indicating that confidence-based gating can actually degrade overall system performance.

Abstract

Deployed vision-language systems often gate their answers on confidence, making confidence robustness relevant to oversight. We study confidence readouts under white-box, image-only attacks constrained to preserve the generated answer byte-identically. Under a reachability assumption, an unmovable readout cannot outperform the answer-string accuracy prior, whose pooled value is 0.617. Independently of that assumption, a uniform amplitude certificate below a measurable threshold guarantees adversarial discrimination above the same floor. Across four vision-language models, three visual question answering benchmarks, five deployed confidence channels and two defense estimators, direct or surrogate-aimed attacks produce itemwise feasible perturbations that refute this uniform certificate in all 84 estimator-by-cell combinations. Coordinated correctness-label-aware attacks drive adversarial discrimination to or below the answer-string floor in all sixty deployed-channel cells, including all fifty-nine that begin above it. Hidden-state interventions and an open-ended text-model activation-space replication show that comparable confidence movement can be induced at the representation level rather than only through adversarial images. None of four tested defense families establishes a robust alternative under the specific evaluation applied to it. In a confidence-gated simulation, a coordinated token-probability attack transferred to a hidden-state gate causes up to 84.8% of previously rejected wrong answers to become accepted. After reweighting to each benchmark's natural correctness prevalence, accepted accuracy falls below the no-gate baseline in eight of twelve cells under transfer and all twelve under a direct gate-aimed attack. Under the studied threat model and budget, confidence is therefore an integrity-sensitive rather than intrinsically robust oversight signal.

Problem

• confidence robustness
• answer-preserving attacks
• vision-language models
• adversarial manipulation
• model oversight

Innovation

• answer-preserving attacks
• confidence robustness
• vision-language models
• adversarial discrimination
• hidden-state interventions