Metadata-Aware Multi-Prompt Reasoning for Zero-Shot Accident Understanding

2026-06-10

AI Summary

This work addresses zero-shot accident understanding in surveillance videos, aiming to accurately localize incidents in time, identify their types, and determine spatial locations. To this end, it proposes the first three-stage pipeline: first, extracting impact event temporal windows based on vision–language similarity; second, introducing a metadata-aware multi-prompt reasoning mechanism that integrates five complementary perspectives and resolves prediction conflicts via an entropy-gated arbitrator; and third, performing open-vocabulary spatial localization with a detector whose outputs are aggregated across keyframes using score-weighted centroids. Evaluated on the zero-shot ACCIDENT@CVPR benchmark, the method significantly outperforms the frame-center baseline, achieving a substantial gain in harmonic mean score, thereby validating the efficacy of the proposed decoupled spatio-temporal–semantic strategy and metadata-driven reasoning.

Abstract

In this paper, we address the problem of zero-shot understanding of accidents from surveillance videos by identifying when an impact event occurs, what type of impact it is, and where in the frame it occurs using natural language. We propose a three-stage pipeline that decomposes the accident understanding into when, what, and where. The first stage extracts a short temporal window around the impact using vision-language similarity. In the second stage, we perform metadata-driven multi-prompt reasoning with five complementary views (baseline, motion, geometry, contrast, and tiebreaker) and resolve disagreement via an entropy-gated pairwise adjudicator. Finally, we localize the impact of an open-vocabulary detector queried on the predicted accident type and scene layout, and aggregate detections across keyframes using a score-weighted centroid. Our pipeline achieves a substantial improvement in the harmonic-mean score over a centre-of-frame baseline on the zero-shot ACCIDENT @ CVPR benchmark. We show that decomposing zero-shot video understanding into temporal localization, semantic classification, and spatial grounding enable more reliable reasoning with vision-language models than direct prompting alone.

Problem

• zero-shot accident understanding
• surveillance videos
• impact event localization
• vision-language reasoning
• spatial-temporal grounding

Innovation

• zero-shot accident understanding
• multi-prompt reasoning
• vision-language models
• temporal localization
• spatial grounding