Prompt-driven Transferable Adversarial Attack on Person Re-Identification with Attribute-aware Textual Inversion

2025-02-27

AI Summary

This work addresses the evaluation of adversarial robustness in person re-identification (re-id) models, identifying a key limitation of current vision-language model (VLM)-based transfer attacks: their overreliance on holistic semantics impedes effective disruption of fine-grained features. To overcome this, we propose an attribute-aware prompt attack. Our method introduces a novel text inversion mechanism that maps pedestrian images to learnable pseudo-tokens, enabling precise perturbation of textual embeddings associated with fine-grained attributes—such as color, pose, and accessories. Leveraging VLMs’ cross-modal alignment capability and contrastive learning, the attack achieves targeted interference of discriminative re-id features. Under cross-model and cross-dataset transfer settings, our approach achieves a 22.9% average Drop Rate improvement over prior methods, establishing new state-of-the-art performance in adversarial evaluation for re-id.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Multiagent Systems: Adversarial Agents

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Person re-identification (re-id) models are vital in security surveillance systems, requiring transferable adversarial attacks to explore the vulnerabilities of them. Recently, vision-language models (VLM) based attacks have shown superior transferability by attacking generalized image and textual features of VLM, but they lack comprehensive feature disruption due to the overemphasis on discriminative semantics in integral representation. In this paper, we introduce the Attribute-aware Prompt Attack (AP-Attack), a novel method that leverages VLM's image-text alignment capability to explicitly disrupt fine-grained semantic features of pedestrian images by destroying attribute-specific textual embeddings. To obtain personalized textual descriptions for individual attributes, textual inversion networks are designed to map pedestrian images to pseudo tokens that represent semantic embeddings, trained in the contrastive learning manner with images and a predefined prompt template that explicitly describes the pedestrian attributes. Inverted benign and adversarial fine-grained textual semantics facilitate attacker in effectively conducting thorough disruptions, enhancing the transferability of adversarial examples. Extensive experiments show that AP-Attack achieves state-of-the-art transferability, significantly outperforming previous methods by 22.9% on mean Drop Rate in cross-model&dataset attack scenarios.

Problem

• Transferable adversarial attacks on person re-identification
• Disrupt fine-grained semantic features of pedestrian images
• Enhance transferability of adversarial examples across models and datasets

Innovation

• Vision-language model alignment
• Attribute-specific textual embedding disruption
• Textual inversion for semantic mapping