A General-Purpose VLM Can Teach an Astronomy Foundation Model to Better Recognize Galaxy Morphology

2026-08-03

AI Summary

This work addresses the challenge of high annotation costs in future large-scale sky surveys, where existing astronomical foundation models struggle to adapt efficiently to new survey data. We propose a novel weakly supervised training framework that leverages the visual-semantic priors of general-purpose vision-language models (VLMs) for galaxy morphology classification. By integrating visual question answering (VQA) benchmarking, uncertainty modeling, and knowledge distillation, our approach guides the domain-specific model Zoobot to achieve cross-survey domain adaptation with limited labeled data. Experimental results demonstrate that this method significantly enhances Zoobot’s classification performance across multiple survey datasets, offering a label-efficient solution well-suited for next-generation surveys such as LSST and Roman.

Abstract

Existing astronomy foundation models provide strong galaxy representations, but adapting them to new survey conditions and survey-specific morphology recognition tasks still requires substantial human supervision. We show that VLM-based VQA systems contain meaningful visual-semantic priors that can serve as weak supervision for downstream morphology classifiers and improve morphology classification under limited human-label budgets. We first introduce a survey-oriented VQA benchmark spanning two representative imaging regimes and evaluate state-of-the-art VLMs on galaxy morphology questions. The results show that these models capture useful morphology signals and informative uncertainty, but are not sufficiently reliable to replace human annotators. Motivated by this finding, we use a general-purpose VLM as a morphology teacher for Zoobot, an astronomy foundation model pretrained on large-scale Galaxy Zoo annotations. Across two survey domains and multiple annotation budgets, the VLM teacher consistently improves Zoobot's downstream morphology classification. These results demonstrate that a general-purpose VLM provides knowledge complementary to an astronomy foundation model and can teach it to better recognize galaxy morphology under limited human supervision. The resulting pipeline is designed for label-efficient adaptation to forthcoming large-scale surveys, including the Vera C. Rubin Observatory's Legacy Survey of Space and Time (LSST) and the Nancy Grace Roman Space Telescope. The benchmark and code are publicly available at https://github.com/fw-ic/VLM-morphology-teacher.

Problem

• galaxy morphology
• foundation model
• weak supervision
• label efficiency
• astronomy

Innovation

• vision-language model
• weak supervision
• galaxy morphology
• foundation model
• label-efficient learning