VCR-Bench: A Modular Open-Source Benchmark for Video Classification Robustness

2026-10-06

AI Summary

This study addresses the absence of a unified benchmark and the fragmented, irreproducible implementations in video classification robustness research. To this end, it proposes the first standardized evaluation protocol and modular open-source framework for video robustness. By standardizing video loading, attack-defense execution, and evaluation pipelines, the framework effectively resolves the challenges of robustness assessment along the temporal dimension. Through configuration presets, perceptual metric aggregation, and extensible component interfaces, it integrates 30 models, 14 attack methods, and 10 defense strategies. Systematic evaluations on Kinetics-400 are conducted in terms of accuracy, attack success rate, and computational overhead. This work bridges a critical gap in the field and provides fully reproducible code.

Abstract

Robustness of image classification has several benchmarks, but their video counterparts are absent. In video classification temporal dimension introduces additional degrees of freedom for adversarial attacks, defenses, and preprocessing. Temporal sampling, perturbation budgets, and metric aggregation also interact in ways with no direct analogue in the image setting. Therefore, robustness for video classifiers is studied across scattered, incompatible implementations, making reported numbers hard to reproduce and analyze. We introduce VCR-Bench, a modular open-source benchmark framework that standardizes video loading, wrappers for classifiers, adversarial attacks and defenses, perceptual metrics, configuration presets, and result logging. VCR-Bench currently integrates 30 video classification models, 14 adversarial attacks, and 10 defense wrappers under a common evaluation protocol. We evaluate representative video classifiers, attacks, and defenses on Kinetics-400 subset, reporting clean accuracy, attack success rate, perceptual quality, runtime, and memory usage. VCR-Bench is released with documented installation, reproducible run presets, component-extension interfaces, and scripts for reproducing the reported results at https://github.com/msu-video-group/vcr-bench.

Problem

• Video Classification
• Robustness Benchmark
• Adversarial Attacks
• Temporal Dimension
• Reproducibility

Innovation

• Video Classification Robustness
• Adversarial Attacks
• Modular Benchmark
• Open-Source Framework
• Temporal Dimension