Rethinking Test-Time Training: Tilting The Latent Distribution For Few-Shot Source-Free Adaptation

2026-02-02

AI Summary

This work proposes a training-free inference method for few-shot test-time adaptation under the challenging setting where the model is entirely frozen and source data are inaccessible. The approach performs a measure transformation on the encoder’s latent embedding distribution via exponential tilting, leveraging the support set to compute task similarity scores and reweighting the distribution according to a KL-optimal criterion to adapt to the new task. As the first method to achieve training-free test-time adaptation with a completely frozen model, it relies solely on inference-stage distribution correction. Extensive experiments demonstrate that the proposed method matches or even surpasses the performance of approaches requiring parameter updates across multiple few-shot benchmarks, thereby validating the effectiveness and potential of purely inference-level adaptation.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Search and Optimization: Learning to Search
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Often, constraints arise in deployment settings where even lightweight parameter updates e.g. parameter-efficient fine-tuning could induce model shift or tuning instability. We study test-time adaptation of foundation models for few-shot classification under a completely frozen-model regime, where additionally, no upstream data are accessible. We propose arguably the first training-free inference method that adapts predictions to the new task by performing a change of measure over the latent embedding distribution induced by the encoder. Using task-similarity scores derived from a small labeled support set, exponential tilting reweights latent distributions in a KL-optimal manner without modifying model parameters. Empirically, the method consistently competes with parameter-update-based methods across multiple benchmarks and shot regimes, while operating under strictly and universally stronger constraints. These results demonstrate the viability of inference-level distributional correction for test-time adaptation even with a fully-frozen model pipeline.

Problem

• test-time adaptation
• few-shot classification
• frozen model
• source-free
• latent distribution

Innovation

• test-time adaptation
• frozen model
• exponential tilting
• latent distribution
• few-shot classification