A Semi-Supervised Learning Method for the Identification of Bad Exposures in Large Imaging Surveys

2025-07-17

AI Summary

To address the low efficiency of manual image quality assessment in large-scale astronomical surveys (e.g., DECaLS), this paper proposes a semi-supervised anomaly detection method leveraging Vision Transformers (ViT) and a k-nearest neighbors (kNN) classifier. The approach employs self-supervised ViT pretraining followed by fine-tuning with minimal labeled samples to identify defective exposures in low-extinction regions of DECam imaging data. Its key innovation lies in integrating self-supervised representation learning with a lightweight, non-parametric kNN classifier, complemented by clustering-space analysis to validate discriminative capability. Evaluated on DECaLS Data Release 11, the method successfully identified 780 low-quality exposures; manual verification confirmed high precision. This framework significantly enhances the efficiency, scalability, and practicality of automated quality control in astronomical data processing pipelines.

Technology Category

Computer Vision: Adversarial Attacks & RobustnessData Mining & Knowledge Management: Anomaly/Outlier DetectionMachine Learning: Calibration & Uncertainty Quantification

Application Category

Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasetsSearch and Retrieval-Augmented AI: Vertical and domain-specific searchGraph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

As the data volume of astronomical imaging surveys rapidly increases, traditional methods for image anomaly detection, such as visual inspection by human experts, are becoming impractical. We introduce a machine-learning-based approach to detect poor-quality exposures in large imaging surveys, with a focus on the DECam Legacy Survey (DECaLS) in regions of low extinction (i.e., $E(B-V)<0.04$). Our semi-supervised pipeline integrates a vision transformer (ViT), trained via self-supervised learning (SSL), with a k-Nearest Neighbor (kNN) classifier. We train and validate our pipeline using a small set of labeled exposures observed by surveys with the Dark Energy Camera (DECam). A clustering-space analysis of where our pipeline places images labeled in ``good'' and ``bad'' categories suggests that our approach can efficiently and accurately determine the quality of exposures. Applied to new imaging being reduced for DECaLS Data Release 11, our pipeline identifies 780 problematic exposures, which we subsequently verify through visual inspection. Being highly efficient and adaptable, our method offers a scalable solution for quality control in other large imaging surveys.

Problem

• Detect poor-quality exposures in large imaging surveys
• Replace impractical human visual inspection methods
• Provide scalable quality control for astronomical surveys

Innovation

• Semi-supervised learning for image anomaly detection
• Vision transformer trained via self-supervised learning
• k-Nearest Neighbor classifier for exposure quality