🤖 AI Summary
This study investigates whether vision-language models (VLMs) can acquire concept-level abstraction capabilities without explicit high-level semantic supervision. To address this, we propose a group-wise contrastive learning framework tailored for the CLEAR GLASS model. Leveraging our self-constructed MAGIC dataset—comprising semantically grouped image-text pairs—we introduce a novel group-wise contrastive loss that jointly optimizes inter-group discrimination and intra-group alignment, thereby inducing concept-level semantic representations in the latent space. Our method integrates implicit semantic space modeling with unsupervised group induction, eliminating reliance on manually annotated high-level concept labels. Experiments demonstrate that the proposed approach significantly outperforms existing state-of-the-art methods on abstract concept recognition tasks. Notably, it is the first to enable VLMs to consistently and transferably emerge concept abstraction capabilities without any explicit high-level supervisory signals.
📝 Abstract
Humans can recognize an image as an instance of a general concept, beyond simply identifying its objects and their relationships. In this paper, we investigate 1. The extent to which VLMs have this concept abstraction capacity, and 2. Strategies for encoding the sort of higher-concept information in images that would enable the resulting VLM model (CLEAR GLASS model) to have this capability to a greater degree. To this end, we introduce a grouped image-caption dataset (MAGIC), which consists of several groups of image captions and for each group a set of associated images and higher-level conceptual labels. We use a novel contrastive loss technique to induce the model to encode in the representation of each image (caption) in a group the information that is common to all members of the image-caption group. Our main contribution is a grouped contrastive loss function based on text-image contrastive groups (outer contrastive loss) as well as an inner loss which measures the distances between image-caption instances in the group. Our training methodology results in the CLEAR GLASS model having the concept abstraction capacity as an emergent capacity because the model is not exposed to the higher-level concepts associated with each group. Instead, the training forces the model to create for each image-caption group a semantic representation that brings it closer to the semantic representation of the higher-level concepts in the latent semantic space. Our experiments show that this training methodology results in a model which shows improvement in abstract concept recognition compared to SOTA models.