Adapting Vision-Language Models from Iconic to Inclusive for Multi-Label Recognition Without Labels

2026-06-09

AI Summary

This work addresses the limitation of existing vision-language models in multi-label image recognition, where predictions are often dominated by salient objects, hindering comprehensive understanding of co-occurring entities. To overcome this, the authors propose an unsupervised two-stage framework—“Crop” and “Stitch”—that guides a pretrained model toward inclusive multi-label comprehension with only a single training round. The approach introduces a multi-sampling response estimator to suppress over-reliance on dominant objects and incorporates a multi-object fusion adapter to refine label distribution. Experimental results demonstrate that the method significantly outperforms current unsupervised approaches across four public benchmarks and even surpasses several weakly supervised baselines, confirming its effectiveness and generalizability.

Abstract

Understanding multi-label images remains a challenging task in computer vision. With the rapid progress of vision-language multimodal learning, vision-language models (VLMs) enable zero-shot recognition without labeled data. However, due to their intrinsic design, these models often prioritize the most iconic object and omit other contextual positives. This intrinsic bias conflicts with the nature of multi-label learning, thereby limiting their applicability. In this work, we propose an unsupervised framework that adapts VLMs from iconic recognition toward inclusive understanding, enabling label-free multi-label image recognition. Our approach consists of two key stages, ``cutting'' and ``sewing'': In the cutting stage, we present the multi-sampling response estimator to prevent the model from concentrating only on one single object. In the second sewing stage, the multi-object blend adaptation is introduced to adjust the labels to better conform to the multi-label distribution while preserving the intrinsic characteristics of the original model within only one epoch. Extensive experiments show that our framework significantly outperforms existing unsupervised approaches on four public datasets, even surpassing several representative weakly supervised baselines. These results demonstrate the potential of adapting pre-trained VLMs for more comprehensive visual understanding without manual annotations. Our code is publicly available at https://github.com/iCVTEAM/TailorCLIP.

Problem

• multi-label recognition
• vision-language models
• zero-shot learning
• label-free
• inclusive understanding

Innovation

• vision-language models
• multi-label recognition
• unsupervised learning
• zero-shot learning
• inclusive understanding