When Modalities Fail to Tango: Conformal Backdoor Detection in Multimodal Contrastive Learning

2026-08-04

AI Summary

Existing backdoor detection methods in multimodal contrastive learning rely on CLIPScore, suffering from distributional overlap and the lack of statistical guarantees due to fixed thresholds. This work proposes CASCADE, a novel framework that introduces conformal prediction into multimodal backdoor detection for the first time, providing provable statistical confidence bounds. CASCADE employs a two-stage coarse-to-fine mechanism: it first selects high-confidence samples based on cross-modal consistency, then computes an inconsistency score via textual-space similarity to precisely identify poisoned samples within ambiguous regions. Experiments on the CC3M dataset demonstrate that the method achieves an average false positive rate of only 5.79% at 100% true positive rate, with an AUROC of 0.9867, while maintaining robustness against adaptive attacks.

Abstract

Backdoor attacks in multimodal contrastive learning (MCL) have garnered growing attention in recent years, as many downstream tasks critically depend on pre-trained MCL models. Existing detection-based defenses predominantly rely on the CLIPScore metric, under the assumption that poisoned pairs exhibit lower semantic similarity between the image and the caption. However, we identify two critical flaws remaining in existing methods: (1) the substantial overlap between CLIPScore distributions of benign and poisoned pairs undermines the reliability of this metric, and (2) fixed-threshold detection cannot provide statistical guarantees for ambiguous samples within overlapping regions. To overcome these limitations, we propose integrating conformal prediction (CP), a statistical framework that quantifies uncertainty through nonconformity scores (NCSs), to establish provable confidence bounds for detecting poisoned image-caption pairs. Building on CP, we introduce CASCADE, a novel two-stage Coarse-to-Fine Conformal Backdoor Detection framework. The coarse-grained stage uses cross-modality consistency to identify high-confidence benign and poisoned pairs. In the fine-grained stage, a reference set is constructed from high-confidence poisoned pairs, and instance-level NCSs based on text-space similarity are computed for each sample in the unidentified subset. These NCSs measure conformity to the poisoning distribution and enable precise identification of latent poisoned pairs within the unidentified subset. Extensive experiments on the large-scale CC3M dataset demonstrate that CASCADE achieves an average FPR of 5.79% at 100% TPR and an average AUROC of 0.9867 across diverse attacks, while remaining effective against adaptive attacks.

Problem

• backdoor detection
• multimodal contrastive learning
• CLIPScore
• distribution overlap
• statistical guarantee

Innovation

• Conformal Prediction
• Multimodal Contrastive Learning
• Backdoor Detection
• Nonconformity Score
• CASCADE