Securing Self-supervised Data Curation for Foundation Models Robustness

2026-06-08

AI Summary

Although self-supervised data curation can enhance the generalization of foundation models, it is vulnerable to poisoning attacks from anonymous external data sources. To address this threat, this work proposes a Poisoning Data Detector (PDD) that performs security screening of curated data prior to model training. PDD innovatively integrates the pretrained multimodal model ImageBind with conventional classifiers—such as support vector machines (SVMs) and random forests—to construct an efficient detection system. This architecture also enables rapid extension to detect emerging attack types through ensemble methods. Evaluated on a dataset comprising 176,200 images and three categories of adversarial attacks, SVM-based PDD achieves state-of-the-art performance on both in-distribution and out-of-distribution data.

Abstract

Self-supervised data curation provides a pathway to scaling and improving the generalization capabilities of machine learning models. By leveraging self-supervised learning (SSL) for data curation, the demand for massive training datasets required by foundation models can be effectively met. SSL greatly alleviates the costs associated with annotation and manual dataset curation while minimizing the need for human oversight. However, the integrity of SSL-curated datasets must be rigorously checked, as reliance on anonymous and unvetted external sources can substantially increase the risk of data poisoning. In this paper, we propose a Poisoned Data Detector (PDD), an active defense mechanism designed to ensure the integrity of SSL-curated datasets prior to foundation model training. PDDs are designed using a combination of the pretrained ImageBind model and traditional classifiers, including Random Forest (RF), k-Nearest Neighbors (KNN), Naive Bayes (NB), and Support Vector Machines (SVM). We rigorously evaluated PDDs using 176,200 images from three diverse datasets and three different adversarial attacks encompassing both in-distribution and out-of-distribution scenarios. Notably, SVM-PDD achieves superior performance for both in-distribution (Set3-Set5) and out-of-distribution (TrueFace and 140K RealFace) datasets. Our design demonstrates strong scalability and enables the rapid integration of new adversarial attack detectors through an ensemble approach.

Problem

• self-supervised data curation
• foundation models
• data poisoning
• data integrity
• adversarial attacks

Innovation

• Poisoned Data Detection
• Self-supervised Learning
• Foundation Models
• Data Curation
• Adversarial Robustness