Test-Time Adaptation via Dual Distillation for Videos Under Severe Distribution Shifts

2026-07-27

AI Summary

This work addresses the significant performance degradation of video models under severe distribution shifts by proposing TADD, an online test-time adaptation framework tailored for continuous temporal data. Built upon a frozen CLIP backbone, TADD introduces a lightweight learnable projection adapter and innovatively combines zero-shot distillation and target distillation losses. During inference, only the adapter parameters are updated, enabling efficient and dynamic domain adaptation without modifying the pretrained features. Extensive experiments demonstrate that TADD substantially outperforms existing methods on three video action recognition benchmarks—UCF-HMDB, Daily-DA, and Sports-DA—achieving absolute accuracy gains of up to 3.81%, 2.63%, and 3.03%, respectively.

Abstract

Deep learning models have achieved state-of-the-art performance in several computer vision tasks. However, they experience severe performance degradation when applied to real-world scenarios due to unanticipated distribution shifts. Test-Time Adaptation (TTA) attempts to solve this problem by using unlabeled data from the target domain to dynamically adapt to the test distribution at inference time, without access to the source data. However, TTA remains a challenging problem when adapting to continuous, temporally correlated data, such as videos, and in scenarios where the target domain contains severe domain shifts. For this reason, few works in the literature explore TTA for videos under such extreme conditions. To overcome these limitations, we propose Test-time Adaptation via Dual Distillation (TADD), an online adaptation framework that relies on a lightweight projection adapter to bridge the domain gap. The adapter module is pre-trained on the source domain and then adapted to the target using our proposed complementary losses: (i) zero-shot distillation, which encourages alignment with the domain-agnostic features from a pre-trained vision-language model (VLM); and (ii) target distillation, which retains the source domain discriminative knowledge encoded in the pre-trained adapter. Built upon a frozen CLIP backbone, our method introduces this lightweight projection adapter as the sole updatable component during inference. We conducted extensive evaluations on three well-known video action recognition benchmarks: UCF-HMDB, Daily-DA, and Sports-DA. Our experiments in the closed-set scenario demonstrate that our method consistently outperforms state-of-the-art TTA baselines. Notably, our TTA approach improves upon previous methods by up to +3.81% on UCF-HMDB, +2.63% on Daily-DA, and +3.03% on Sports-DA.

Problem

• Test-Time Adaptation
• Distribution Shifts
• Video Action Recognition
• Domain Adaptation
• Temporal Data

Innovation

• Test-Time Adaptation
• Dual Distillation
• Video Domain Shift
• Projection Adapter
• Vision-Language Model