Backdoor Attacks on Open Vocabulary Object Detectors via Multi-Modal Prompt Tuning

2025-11-16

AI Summary

This work identifies a novel backdoor threat against open-vocabulary object detectors (OVODs) under multimodal prompt tuning. Unlike prior studies, it explicitly addresses the security vulnerability at the prompt layer—a previously overlooked aspect. We propose the first stealthy prompt-tuning-oriented backdoor attack: jointly optimizing learnable text prompts and visual triggers, embedding small-patch triggers via image-text co-tuning; and incorporating a curriculum learning strategy to progressively shrink trigger size for efficient activation. The attack requires no model weight modification and supports both misclassification and object-removal objectives. Evaluated on COCO and LVIS, it achieves >95% attack success rates while simultaneously improving downstream detection performance on clean samples. Our approach provides a new perspective and practical tool for security assessment of multimodal foundation models.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Open-vocabulary object detectors (OVODs) unify vision and language to detect arbitrary object categories based on text prompts, enabling strong zero-shot generalization to novel concepts. As these models gain traction in high-stakes applications such as robotics, autonomous driving, and surveillance, understanding their security risks becomes crucial. In this work, we conduct the first study of backdoor attacks on OVODs and reveal a new attack surface introduced by prompt tuning. We propose TrAP (Trigger-Aware Prompt tuning), a multi-modal backdoor injection strategy that jointly optimizes prompt parameters in both image and text modalities along with visual triggers. TrAP enables the attacker to implant malicious behavior using lightweight, learnable prompt tokens without retraining the base model weights, thus preserving generalization while embedding a hidden backdoor. We adopt a curriculum-based training strategy that progressively shrinks the trigger size, enabling effective backdoor activation using small trigger patches at inference. Experiments across multiple datasets show that TrAP achieves high attack success rates for both object misclassification and object disappearance attacks, while also improving clean image performance on downstream datasets compared to the zero-shot setting.

Problem

• Investigating backdoor attack vulnerabilities in open-vocabulary object detectors
• Developing multi-modal prompt tuning method to implant hidden malicious behavior
• Enabling effective backdoor activation with small trigger patches during inference

Innovation

• Multi-modal prompt tuning for backdoor injection
• Joint optimization of image and text prompts
• Curriculum training with shrinking trigger size