Image Classifiers are Efficient Self-Supervised Video Representation Learners

2026-09-30

AI Summary

This study addresses the computational inefficiency in video self-supervised learning caused by reliance on heavy 3D architectures or reconstruction-based decoders. To this end, it proposes VideoMSN, a framework that treats videos as "super-images" and repurposes image foundation models by reusing Vision Transformer (ViT) architectures with DINO-v3/DeiT-v3 pretrained weights. Through a masked siamese network, VideoMSN aligns spatiotemporal embeddings, enabling efficient representation learning without requiring a decoder. The proposed method achieves state-of-the-art performance on benchmarks such as Kinetics while reducing the number of pretraining epochs by 160 times. Furthermore, it demonstrates strong capabilities in few-shot classification tasks. Overall, this work significantly lowers the computational cost of video representation learning by effectively leveraging pretrained image models for spatiotemporal feature extraction.

Abstract

We introduce VideoMSN, a Masked Siamese Network framework for efficient self-supervised spatio-temporal representation learning in videos. Instead of relying on heavy 3D architectures or reconstruction-based autoencoders for learning with unlabeled data, we repurpose standard image Vision Transformers by representing videos as super images which are grids composed of frames sampled from videos. From each super image, we construct two views: one with spatial patch masking and the other with temporal frame masking, ensuring no information leakage across frames. A shared Vision Transformer (ViT) encoder aligns their embeddings using a masked Siamese loss, capturing both motion and appearance cues without reconstruction. Our decoder-free formulation leverages an image foundation model towards efficient video representation learning. Starting from pretrained DINO-v3 and DeiT-v3 image encoders, VideoMSN achieves state-of-the-art performance on Kinetics-400, UCF101, and HMDB51 while requiring up to $32\times$ fewer and $160\times$ fewer video pretraining epochs compared to prior video self-supervised learning methods. Our proposed approach also shows strong performance in low-shot classification, confirming the transferability of the learned representations in a label-scarce scenario. Project Page: https://cvir.github.io/projects/videomsn.

Problem

• self-supervised video representation learning
• spatio-temporal representation
• computational efficiency
• Vision Transformer

Innovation

• Masked Siamese Network
• Self-Supervised Video Representation Learning
• Vision Transformer
• Super Image
• Decoder-Free