GoldiCLIP: The Goldilocks Approach for Balancing Explicit Supervision for Language-Image Pretraining

2026-03-25

AI Summary

This work addresses the data inefficiency of large-scale vision-language models, which typically rely on massive datasets that hinder the exploration of more effective supervisory signals. To this end, the authors propose GoldiCLIP, a framework that achieves highly efficient pretraining using only 30 million images—300 times fewer than mainstream approaches. GoldiCLIP integrates text-conditioned self-distillation, an encoder-decoder architecture trained with visual question answering (VQA) objectives, and an uncertainty-based automatic weighting mechanism for multi-task losses, jointly optimizing feature alignment and generalization. Experimental results demonstrate that GoldiCLIP outperforms the strongest baselines by 2.2, 2.0, and 5.9 percentage points on MSCOCO retrieval, fine-grained retrieval, and query-focused retrieval tasks, respectively, matching the performance of models trained on billion-scale datasets.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Web Mining and Content Analysis: Large pretrained models with web data
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Until recently, the success of large-scale vision-language models (VLMs) has primarily relied on billion-sample datasets, posing a significant barrier to progress. Latest works have begun to close this gap by improving supervision quality, but each addresses only a subset of the weaknesses in contrastive pretraining. We present GoldiCLIP, a framework built on a Goldilocks principle of finding the right balance of supervision signals. Our multifaceted training framework synergistically combines three key innovations: (1) a text-conditioned self-distillation method to align both text-agnostic and text-conditioned features; (2) an encoder integrated decoder with Visual Question Answering (VQA) objective that enables the encoder to generalize beyond the caption-like queries; and (3) an uncertainty-based weighting mechanism that automatically balances all heterogeneous losses. Trained on just 30 million images, 300x less data than leading methods, GoldiCLIP achieves state-of-the-art among data-efficient approaches, improving over the best comparable baseline by 2.2 points on MSCOCO retrieval, 2.0 on fine-grained retrieval, and 5.9 on question-based retrieval, while remaining competitive with billion-scale models. Project page: https://petsi.uk/goldiclip.

Problem

• vision-language models
• data efficiency
• contrastive pretraining
• supervision quality
• large-scale datasets

Innovation

• self-distillation
• vision-language pretraining
• uncertainty-based weighting
• visual question answering
• data-efficient learning