MuST-VAD: Mutual Structured Learning for Video Anomaly Detection

2026-08-07

AI Summary

This work addresses the performance limitations imposed by fixed features in weakly supervised video anomaly detection by proposing a bidirectional mutual learning framework. For the first time, it extends unidirectional feature transfer to a reciprocal knowledge exchange between large vision-language models and anomaly detectors. The framework enables efficient co-training through alternating updates, critical clip selection, and a confidence-weighted reliable supervision mechanism. Integrating weakly supervised learning, bidirectional knowledge distillation, and an annotation-guided question-answering strategy, the method achieves an AUROC of 88.63% and an average precision of 42.46% on the UCF-Crime dataset, outperforming the current state-of-the-art approach by 4.13 percentage points.

Abstract

In this paper, we propose MuST-VAD, a mutual structured learning framework for weakly supervised video anomaly detection (VAD) in which an anomaly detector and a large vision-language model (LVLM) exchange their acquired knowledge. Detectors in weakly supervised VAD learn anomaly scores from features extracted by a fixed, task-agnostic backbone. These fixed features bound the achievable detection accuracy. Recent methods therefore transfer LVLM semantics into the detector as richer features. However, this transfer is one-way: what the detector learns about the target videos never returns to the LVLM. MuST-VAD extends the one-way transfer into a bidirectional learning loop. In this loop, the latest detector predictions supervise the LVLM adaptation, and the adapted LVLM returns updated representations that retrain the detector; the two models alternate these updates over small video groups. Both models train on detector-selected key clips, while confidence weighting and annotation-anchored question answering keep the exchanged supervision reliable. On UCF-Crime, our mutual learning improves the one-pass transfer baseline from 88.15% to 88.63% AUROC and from 37.25% to 42.46% average precision (AP), outperforming the state-of-the-art method in AP by 4.13 points.

Problem

• video anomaly detection
• weakly supervised learning
• vision-language model
• mutual learning
• feature transfer

Innovation

• mutual structured learning
• video anomaly detection
• weakly supervised learning
• vision-language model
• bidirectional knowledge transfer