LFM: Leveraging Foundation Models for Source-Free Universal Domain Adaptation

2026-07-20

AI Summary

This work addresses the challenging problem of universal domain adaptation under unsupervised target data, where both covariate shift and label shift coexist. It proposes the first framework to integrate foundation models into this setting: leveraging a vision-language model to compute similarity scores between target samples and textual class descriptions—including those of unknown classes generated by a large language model—to identify the type of label shift and detect novel categories. Building upon a source-initialized target model, the approach refines pseudo-labels through a consensus mechanism that effectively combines the semantic priors from foundation models with knowledge from the source model. Extensive experiments across diverse label shift configurations and multiple benchmarks demonstrate that the method significantly outperforms existing approaches, validating its effectiveness and superiority.

Abstract

Source-free universal domain adaptation (SF-UniDA) adapts a pre-trained source model to an unlabeled target domain under both covariate and label shifts, without access to source data. However, existing SF-UniDA methods rely on inefficient techniques such as threshold tuning and clustering. Foundation models (FMs), known for their generalization and zero-shot capabilities, remain underexplored in SF-UniDA. In this paper, we propose a framework that leverages foundation models (LFM) for SF-UniDA. We use a vision-language model (VLM) to compute similarities between target samples and text labels, including those for unknown classes generated by prompting a large language model. The label shift type is determined by analyzing the coefficient of variation of a similarity-based sample-level score. Unknown samples are identified using a binary Gaussian mixture model fitted to another similarity-based metric. Under a consensus strategy, the pseudo-labels generated by the VLM are refined by the target model initialized with the pre-trained source model, integrating knowledge from both the source domain and foundation models. Finally, these refined pseudo-labels are used to train the target model. Extensive experiments across all possible label shifts and multiple benchmarks demonstrate the effectiveness and superiority of our proposed LFM framework. Our code is available at https://github.com/iamjingli/LFM.

Problem

• Source-Free Universal Domain Adaptation
• Foundation Models
• Label Shift
• Covariate Shift
• Unknown Class Detection

Innovation

• Foundation Models
• Source-Free Universal Domain Adaptation
• Vision-Language Model
• Label Shift
• Pseudo-Label Refinement