COSMO: Consensus-Driven Shift Modulation for Source-Free Domain Adaptation

2026-08-05

AI Summary

This work addresses the challenge in source-free domain adaptation where self-supervision tends to reinforce source-domain bias, and existing approaches fail to account for sample-level reliability differences when integrating knowledge from vision-language models (VLMs), often overriding valid source evidence. To mitigate this, the paper formulates the problem as a sample-level reliability allocation task and introduces a consensus-driven shift modulation strategy. By establishing an anchored shared consensus mechanism that aligns the source model with the VLM, the method leverages sample-specific initial consensus, dual-branch evidence re-aggregation, and dynamic anchor adjustment based on consensus uncertainty and training progress to enable reliability-aware adaptive fusion. Evaluated on four benchmarks, the approach achieves state-of-the-art performance under the same VLM architecture, effectively preserving source-domain evidence while incorporating complementary semantic information.

Abstract

Source-free domain adaptation (SFDA) adapts a source-trained model to an unlabeled target domain without source data, a practical setting under privacy or storage constraints. Yet its self-generated supervision can reinforce source bias under substantial domain shifts. Pretrained vision-language models (VLMs) offer complementary semantic knowledge, but the relative reliability of the source model and VLM varies across target samples. Existing cross-model guidance does not explicitly account for this variation and may overwrite valid source-derived evidence under conflict, a failure we term source-derived evidence forgetting. We formulate VLM-guided SFDA as a sample-wise reliability-allocation problem and propose Consensus-Driven Shift Modulation (COSMO). COSMO replaces expert-to-expert guidance with co-adaptation through an anchored shared consensus. It first forms a sample-specific initial consensus that favors the more concentrated prediction. During adaptation, COSMO re-aggregates both branches' evolving evidence and regulates how far the resulting consensus moves from its initial anchor based on consensus uncertainty and training progress. This keeps the shared supervision anchored yet adaptive. Across four benchmarks, COSMO achieves state-of-the-art performance under matched VLM backbones. Further analyses indicate that it better balances the retention of valid source-derived evidence with the absorption of complementary VLM evidence.

Problem

• Source-Free Domain Adaptation
• Domain Shift
• Vision-Language Models
• Evidence Forgetting
• Model Reliability

Innovation

• source-free domain adaptation
• vision-language models
• consensus modulation
• evidence retention
• reliability allocation