VidMsg: A Benchmark for Implicit Message Inference in Short Videos

2026-06-02

AI Summary

This work addresses the challenge that existing video understanding methods struggle to capture the implicit intent of creators in short videos. To this end, we introduce VidMsg, the first benchmark for short-video understanding centered on latent messages. Built through a message-first pipeline, VidMsg comprises 400 YouTube videos spanning nine broad themes and 52 fine-grained message categories. By leveraging large language models to generate indirectly expressed scenarios and incorporating human-curated non-explicit videos, the benchmark supports bidirectional message–video retrieval and multiple-choice question answering. We also propose VidVec-Msg as a baseline method. Experimental results reveal that state-of-the-art models perform substantially below human levels, highlighting the critical need for pragmatic reasoning and fine-grained semantic discrimination, and thereby filling a key gap in evaluating implicit intent understanding in video content.

Abstract

Understanding short online videos involves more than identifying visible objects and actions; video makers often include an underlying message or purpose in the clip. We introduce VidMsg, a benchmark for evaluating implicit message understanding in short, internet-native video clips. VidMsg contains 400 YouTube-derived clips across 9 practical topic areas and 52 fine-grained target messages, covering domains such as career and finance, education, health and well-being, culture, safety, sustainability, and lifestyle. VidMsg is constructed through a message-first pipeline: an LLM first translates target messages into indirect search scenarios, which are used to retrieve candidate clips. Human annotators then retain clips that convey the intended message without being overly explicit. VidMsg is designed primarily for bidirectional message-clip retrieval for scalable applications such as video search and recommendation, where systems must capture holistic video understanding. In addition to retrieval, VidMsg includes a diagnostic multiple-choice QA benchmark, where models select the intended message of a clip from semantically related alternatives. Experiments with contemporary video-language and retrieval models show that strong models often fail on VidMsg, because the task requires pragmatic inference, integration of contextual cues, and discrimination among semantically close messages. We also introduce VidVec-Msg, a baseline method that improves message-oriented retrieval while leaving substantial headroom for future work.

Problem

• implicit message inference
• short video understanding
• video-language models
• pragmatic inference
• message-clip retrieval

Innovation

• implicit message inference
• video-language understanding
• message-first pipeline
• bidirectional retrieval
• pragmatic video comprehension