HeiCo-FOCUS: A Clinically Grounded Dataset for Long-Context Video Understanding

2026-10-07

AI Summary

This study addresses the lack of long-range temporal consistency evaluation in existing video understanding benchmarks by constructing the first clinically oriented long-context video benchmark. Leveraging hours of colorectal surgery footage, 30,000 question-answer pairs are generated through collaborative crowdsourced and expert annotation, alongside a progressive multi-track evaluation framework designed to probe the limits of models' temporal reasoning. Experiments reveal that most state-of-the-art models perform only marginally better than text-only baselines, with temporal localization accuracy as low as 19.7%. These findings expose significant deficiencies in current vision-language models regarding sustained tracking and fine-grained temporal comprehension over extended videos.

Abstract

Recent advances in Vision-Language Models (VLMs) have led to rapid progress in video understanding across a wide range of benchmark tasks. However, existing evaluations largely focus on short-term reasoning, failing to assess a critical capability: maintaining cumulative temporal consistency over extended time horizons. To close this evaluation gap, we introduce HeiCo-FOCUS, a clinically grounded dataset for evaluating long-context video understanding through the task of Foreign Object Contextual Understanding in Surgery. Built on a dataset of Heidelberg Colorectal surgeries, this task requires models to continuously track multiple objects as they are inserted, manipulated, occluded, and removed over procedures lasting up to hours. HeiCo-FOCUS comprises 30,000 visual question answering (VQA) pairs covering five core capabilities: object recognition, temporal grounding, aggregation, event and procedural understanding, and complex reasoning. The dataset was constructed through a rigorous multi-stage annotation pipeline involving large-scale crowd annotation and 39 surgical domain experts to ensure high quality and clinical relevance. To systematically probe model behavior, we introduce a multi-track evaluation framework that progressively increases temporal and contextual demands from single frames to full procedures. Experiments with ten frontier VLMs show that HeiCo-FOCUS tasks are far from solved: only around half of the models clearly outperform a text-only baseline. Across the video tracks, models perform best on event and procedural understanding (mean Accuracy: 56.5% across all models), while temporal grounding remains particularly challenging for all evaluated models (mean Accuracy: 19.7%). We therefore expect HeiCo-FOCUS to serve as a catalyst for the development of models capable of reliable, temporally consistent reasoning over hours-long videos.

Problem

• long-context video understanding
• temporal consistency
• vision-language models
• video evaluation
• temporal grounding

Innovation

• Long-context video understanding
• Vision-Language Models
• Clinical dataset
• Visual question answering
• Multi-track evaluation