🤖 AI Summary
This work addresses the challenges of semantic-temporal misalignment and insufficient multi-scale temporal modeling in few-shot action recognition by proposing the STAR framework, which enforces fine-grained semantic-temporal consistency through two core modules: semantic alignment and temporal awareness. The framework innovatively integrates Temporal Semantic Attention (TSA) and a semantics-guided Mamba block, jointly optimized with temporal dependency descriptors generated by a large language model to align semantic and dynamic temporal representations. Additionally, a Semantic-Temporal Prototype Refiner (STPR) and a multi-frequency temporal sampling strategy are introduced to enhance cross-video temporal modeling. Extensive experiments demonstrate that STAR significantly outperforms existing methods across five benchmarks, achieving absolute gains of 8.1%, 6.7%, and 7.3% under the 1-shot setting on SSv2-Full, SSv2-Small, and HMDB51, respectively.
📝 Abstract
Few-shot action recognition (FSAR) requires models to generalize to novel action categories from only a handful of annotated samples. Despite progress with vision-language models, existing approaches still suffer from semantic-temporal misalignment, where static textual prompts fail to capture decisive visual cues that appear sparsely across sequences, and from inadequate modeling of multi-scale temporal dynamics, as short-term discriminative cues and long-range dependencies are often either oversmoothed or fragmented. To address these challenges, we propose Semantic Temporal Adaptive Representation Learning (STAR), a unified framework, consisting of a semantic-alignment component and a temporal-aware component, effectively bridging the semantic and temporal gaps and transferring the sequence modeling capability of Mamba into the FSAR. The semantic alignment module introduces a Temporal Semantic Attention (TSA) mechanism, which performs frame-level cross-modal alignment with textual cues, ensuring fine-grained semantic-temporal consistency. The temporal-aware module incorporates a Semantic Temporal Prototype Refiner (STPR) that integrates semantic-guided Mamba blocks with multi-frequency temporal sampling and bidirectional state-space refinement, yielding semantically aligned prototypes with enhanced discriminative fidelity and temporal consistency. Furthermore, temporally dependent class descriptors derived from large language models (LLMs) provide long-range semantic guidance. Extensive experiments on five FSAR benchmarks demonstrate the consistent superiority of STAR over state-of-the-art methods. For instance, STAR achieves up to 8.1% and 6.7% gains on the SSv2-Full and SSv2-Small datasets under the 1-shot setting, and 7.3% on HMDB51, validating its effectiveness under limited supervision. The code is available at https://github.com/HongliLiu1/STAR-main.