Efficient Video Dataset Distillation via Cluster-Guided Prototype Blending

2026-08-04

AI Summary

This work addresses the high computational cost of temporal optimization and the difficulty of preserving intra-class diversity in large-scale video dataset distillation. To this end, the authors propose ProtoBlend, a novel framework that achieves gradient-free, constructive video distillation for the first time. ProtoBlend integrates teacher-guided temporal segment selection, feature clustering–driven prototype assignment, and a linear blending strategy between prototypes and anchor frames, effectively balancing temporal representativeness, intra-class diversity, and sample information density. Extensive experiments on four action recognition benchmarks demonstrate that ProtoBlend attains competitive or superior accuracy–efficiency trade-offs compared to existing methods—all without requiring any optimization of the distilled videos themselves.

Abstract

Video dataset distillation aims to compress a large video dataset into a compact surrogate set that preserves its training utility. Most existing approaches synthesize condensed videos through iterative optimization, whose cost is amplified by the temporal dimension. Rather than further reducing the number of optimized variables, we investigate whether effective distilled videos can be constructed without gradient-based optimization of the stored videos. Such a construction-based approach must address three challenges: selecting informative temporal segments, covering diverse intra-class variations under a limited videos-per-class budget, and increasing the information carried by each stored sample. To this end, we propose ProtoBlend, an efficient select-allocate-blend framework. First, teacher-guided temporal clip selection retains a high-confidence segment from each source video. Second, cluster-guided prototype allocation partitions the selected clips in the teacher feature space and assigns one distilled slot to each intra-class cluster. Third, each prototype is blended with an in-cluster anchor, while their teacher predictions are combined using the same coefficient to provide mixture-source supervision. Experiments on four trimmed action-recognition benchmarks demonstrate that ProtoBlend achieves a competitive accuracy-efficiency trade-off without iterative optimization of the distilled videos.

Problem

• video dataset distillation
• temporal segment selection
• intra-class variation
• data efficiency
• gradient-free construction

Innovation

• video dataset distillation
• prototype blending
• cluster-guided allocation
• optimization-free
• temporal clip selection