When few labeled target data suffice: a theory of semi-supervised domain adaptation via fine-tuning from multiple adaptive starts

2025-07-19

AI Summary

Semi-supervised domain adaptation (SSDA) suffers from insufficient theoretical foundations and weak robustness to distribution shifts when only a limited number of labeled samples are available in the target domain. Method: We propose the first structural causal model (SCM)-based theoretical framework for SSDA, quantifying performance bounds under finite target labels and proving minimax optimality of our approach. We design tailored fine-tuning strategies for three fundamental distribution shift types—covariate shift, concept shift, and generalized shift—and introduce Multi-Start Adaptive Fine-Tuning (MASFT), a shift-agnostic algorithm that achieves near-optimal performance without prior knowledge of the shift type. MASFT integrates an unsupervised domain adaptation backbone, multi-start fine-tuning, and a small validation set–driven model selection mechanism. Results: Experiments demonstrate that MASFT significantly reduces reliance on target-domain annotations across diverse shift scenarios, achieving both theoretical rigor and strong empirical robustness.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Multiagent Systems: Adversarial Agents
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Semi-supervised domain adaptation (SSDA) aims to achieve high predictive performance in the target domain with limited labeled target data by exploiting abundant source and unlabeled target data. Despite its significance in numerous applications, theory on the effectiveness of SSDA remains largely unexplored, particularly in scenarios involving various types of source-target distributional shifts. In this work, we develop a theoretical framework based on structural causal models (SCMs) which allows us to analyze and quantify the performance of SSDA methods when labeled target data is limited. Within this framework, we introduce three SSDA methods, each having a fine-tuning strategy tailored to a distinct assumption about the source and target relationship. Under each assumption, we demonstrate how extending an unsupervised domain adaptation (UDA) method to SSDA can achieve minimax-optimal target performance with limited target labels. When the relationship between source and target data is only vaguely known -- a common practical concern -- we propose the Multi Adaptive-Start Fine-Tuning (MASFT) algorithm, which fine-tunes UDA models from multiple starting points and selects the best-performing one based on a small hold-out target validation dataset. Combined with model selection guarantees, MASFT achieves near-optimal target predictive performance across a broad range of types of distributional shifts while significantly reducing the need for labeled target data. We empirically validate the effectiveness of our proposed methods through simulations.

Problem

• Theoretical framework for semi-supervised domain adaptation with limited labeled target data
• Analyzing SSDA performance under various source-target distributional shifts
• Proposing MASFT algorithm to achieve near-optimal performance with minimal labeled data

Innovation

• Structural causal models for SSDA analysis
• Three tailored fine-tuning SSDA methods
• MASFT algorithm for optimal performance