Where Do We (Not) Need Temporal Context in Low-Resource Video Task Adaptation?

2026-06-02

AI Summary

This work addresses the challenge of parameter-efficient fine-tuning (PEFT) for video tasks under low-resource conditions, where the optimal allocation of temporal contextual information remains unclear. The study systematically evaluates PEFT methods applied to both image- and video-pretrained models across appearance-, motion-, and spatially dense tasks. Leveraging probing techniques, it analyzes how temporal modeling is distributed across the backbone network, adapter modules, and probes. The findings reveal—for the first time—the varying necessity of temporal context across different video tasks and its substantial impact on adaptation performance. Based on these insights, the paper establishes practical guidelines for designing efficient adaptation strategies tailored to low-data video scenarios.

Abstract

Parameter-efficient fine-tuning (PEFT) and probing enable adaptation of foundation models using only a small number of trainable parameters, making it attractive for video understanding where annotation and computation are expensive. However, video PEFT has focused on adapting image-pretrained models, while standard PEFT methods can also be applied to video representations. These settings are rarely compared and both confine temporal reasoning to a single component of the model, leaving open how temporal context should be distributed across backbone, PEFT and probe. In this work we provide a systematic study of model adaptation strategies for video understanding. We evaluate methods across appearance-focused, motion-focused and spatially dense settings, with a particular focus on scenarios with limited data where parameter-efficiency is most beneficial. Our results provide new insights into PEFT and probing across settings and demonstrate the importance of temporal context allocation for effective video adaptation

Problem

• temporal context
• low-resource
• video adaptation
• parameter-efficient fine-tuning
• model adaptation

Innovation

• temporal context allocation
• parameter-efficient fine-tuning
• video adaptation
• low-resource learning
• foundation model probing