Corrupting Attention: Evasion-Based Adversarial Attacks on Encoder Attention in Detection Transformers

2026-08-06

AI Summary

Existing adversarial attacks struggle to effectively disrupt the encoder attention mechanisms of detection Transformers. This work proposes a novel ℓ∞-constrained adversarial attack that, for the first time, directly optimizes encoder attention objectives by guiding gradients to steer model attention toward perturbed regions, thereby impairing its spatial reasoning capability. The method is agnostic to specific target formulations and supports diverse attention disruption strategies—including dispersion, reordering, permutation, and peak suppression—while remaining compatible with general attention architectures. Evaluated on COCO, it reduces the mAP of DETR-R50 and DINO-Swin-L to 0.97 and 1.44, respectively, substantially outperforming the strongest existing attacks and revealing that the attention structure itself constitutes a critical vulnerability in model robustness.

Abstract

Adversarial vulnerabilities remain a major concern for the safe deployment of neural networks, particularly in object detection, a core task embedded in many safety-critical systems. Detection transformers have emerged as leading object detectors, yet their adversarial robustness remains comparatively underexplored. Most existing attacks target the detection output rather than the attention mechanism that makes these models distinctive. In this paper, we introduce the first attack that directly optimizes an encoder-attention objective under an imperceptible, bounded $\ell_\infty$ perturbation. Rather than introducing an attacker-owned sink token through a visible patch, it drives the model's own attention toward a corrupted target. We argue that encoder attention concentrates the model's spatial reasoning, so corrupting it propagates through the detection pipeline more disruptively than perturbing the detection output alone. Our attack reduces DETR-R50 mAP on COCO from 42.1 to 0.97, a $\sim 4\times$ reduction in resulting mAP over the strongest existing attack under an identical perturbation budget and iteration count. We further show that this vulnerability is not specific to a particular corruption objective: across four qualitatively distinct targets, dispersion, re-ranking, permutation, and peak-suppression, detection consistently drops below 3 mAP, suggesting that the weakness arises from disrupting the attention structure itself rather than from any single target. Finally, we demonstrate that the attack generalizes across attention formulations, reducing DINO-Swin-L from 56.8 to 1.44 mAP against 7.3 for the strongest prior attack, establishing state-of-the-art on both dense and deformable attention.

Problem

• adversarial attacks
• object detection
• attention mechanism
• Detection Transformers
• encoder attention

Innovation

• adversarial attack
• attention corruption
• Detection Transformers
• encoder attention
• object detection