Test-Time Adaptation with Online Personalized Energy-Based Cache for Fine-Grained Video Expression Recognition

2026-08-06

AI Summary

This work addresses the challenge of test-time distribution shifts in video-based facial expression recognition caused by individual differences, particularly when target samples are scarce and expression categories are rare, making reliable class prototypes difficult to construct. To this end, the authors propose Energy-Based Caching for Personalization (EB-CaP), an online personalized caching mechanism that dynamically generates class prototypes for each test video using semantic similarities from a pretrained CLIP model—without updating model parameters. The approach employs positive and negative caches regulated by adaptive entropy and diversity gates to manage uncertainty and redundancy during cache updates. EB-CaP is the first to integrate a lightweight energy model with online caching for efficient test-time adaptation, achieving significant performance gains over existing test-time adaptation methods on BioVid, StressID, and BAH datasets while maintaining low computational and memory overhead and enhancing fine-grained recognition accuracy.

Abstract

Facial expression recognition (FER) in videos is challenging because models must identify subtle, temporally evolving affective states that vary across individuals. Although vision-language models provide transferable visual-semantic representations, models trained on subject-independent data often degrade under subject-specific distribution shifts at inference time. Existing test-time adaptation (TTA) methods commonly update model parameters during inference, increasing computational cost and latency. Cache-based methods avoid parameter updates, but they usually require enough target samples to form reliable class prototypes, which is difficult early in adaptation and for rarely observed classes. We introduce Energy-Based Cache Personalization (EB-CaP), a subject-based online TTA method for video FER that generates class-specific prototypes personalized to each target video. EB-CaP uses a lightweight energy-based model to sample prototypes from the current unlabeled video and populate a personalized cache online, without accumulating large amounts of target data or storing diverse source prototypes. Its energy function relies only on pretrained CLIP: similarities between the target video embedding and class text embeddings guide prototype sampling. In parallel, positive and negative caches store reliable and uncertain target embeddings. An adaptive entropy gate controls cache updates according to the evolving confidence distribution, while a diversity gate limits redundant samples. Final predictions combine cache-derived scores with the current CLIP scores. Experiments on BioVid, StressID, and BAH show that EB-CaP outperforms state-of-the-art TTA methods while maintaining low computational and memory overhead. Code is available at https://github.com/MasoumehSharafi/EB-CaP.

Problem

• test-time adaptation
• video facial expression recognition
• distribution shift
• class prototypes
• subject-specific adaptation

Innovation

• Test-Time Adaptation
• Energy-Based Model
• Personalized Cache
• Video Facial Expression Recognition
• CLIP