Exploring Structured Semantic Priors Underlying Diffusion Score for Test-time Adaptation

2025-01-01

AI Summary

This work addresses test-time adaptation (TTA), investigating how structured semantic priors implicitly encoded in diffusion model score functions can enhance the out-of-distribution generalization of pre-trained discriminative models. We first reveal that diffusion scores inherently contain transferable, structured semantic information—previously unrecognized in TTA contexts. To leverage this, we propose a single-step denoising knowledge extraction mechanism, circumventing the high computational cost of multi-step Monte Carlo score estimation. Our approach integrates score-matching-driven semantic prior modeling with lightweight feature distillation, enabling efficient TTA without model fine-tuning or access to source-domain data. Extensive experiments on diverse image classification and dense prediction benchmarks demonstrate substantial improvements over state-of-the-art TTA methods. Ablation studies comprehensively validate the effectiveness and necessity of each component, confirming that structured semantic priors from diffusion scores significantly boost robustness under distribution shift.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Capitalizing on the complementary advantages of generative and discriminative models has always been a compelling vision in machine learning, backed by a growing body of research. This work discloses the hidden semantic structure within score-based generative models, unveiling their potential as effective discriminative priors. Inspired by our theoretical findings, we propose DUSA to exploit the structured semantic priors underlying diffusion score to facilitate the test-time adaptation of image classifiers or dense predictors. Notably, DUSA extracts knowledge from a single timestep of denoising diffusion, lifting the curse of Monte Carlo-based likelihood estimation over timesteps. We demonstrate the efficacy of our DUSA in adapting a wide variety of competitive pre-trained discriminative models on diverse test-time scenarios. Additionally, a thorough ablation study is conducted to dissect the pivotal elements in DUSA. Code is publicly available at https://github.com/BIT-DA/DUSA.

Problem

• Semantic Information
• Diffusion Models
• Performance Enhancement

Innovation

• DUSA
• Single-pass Denoising Diffusion
• Adaptive Semantic Information Extraction