FastBench: Can Streaming VLMs Perceive High-Dynamic Real-World Streams?

2026-10-08

AI Summary

This study addresses the neglect of high-dynamic scenes in existing video benchmarks, which hinders Vision-Language Models (VLMs) from balancing spatiotemporal information and capturing rapid events. We construct the first real-world high-dynamic video evaluation benchmark, employing trajectory-guided data generation combined with SAM3 and CoTracker3 for automated verification and iterative manual review. Furthermore, we propose ProactiveFrame, a training-free method that optimizes perception through dual-level sliding windows and text-token-driven active frame rate modulation. Experiments reveal that even the strongest model achieves only 50.7% accuracy, as dense sampling remains constrained by context compression. Notably, ProactiveFrame significantly outperforms uniform sparse sampling, exposing critical deficiencies in current VLMs regarding fine-grained autonomous perception.

Abstract

Streaming Video Large Language Models (VLMs) enable continuous video understanding, yet existing benchmarks focus on low-dynamic scenarios. Under bounded context budgets, models must balance temporal history, spatial resolution, and temporal granularity; sparse sampling at 1--2 FPS misses fast events. We introduce FastBench to evaluate high-dynamic perception in real-world video streams. Its trajectory-grounded pipeline combines QA generation from high-FPS clips, filtering of questions answerable at 2 FPS, answer verification using SAM3 and CoTracker3 trajectories, and three rounds of human inspection. FastBench contains 306 QA pairs across eight domains, six capabilities, and forward, instant, and backward temporal scopes, with human-annotated evidence intervals. We also present ProactiveFrame, a training-free baseline that adjusts incoming frame rates through text tokens. A dual-tier sliding window retains recent high-FPS observations while downsampling older ones into sparse history. Experiments reveal substantial limitations: the strongest model, Gemini-3.5-Flash, scores only 50.7%. Denser sampling improves Qwen3-VL-8B from 32.9% at 2 FPS to 44.6% at 24 FPS, but gains saturate as history is compressed. ProactiveFrame outperforms sparse uniform sampling by 5.4 and 1.5 percentage points, yet remains well below oracle-guided focusing, showing that current VLMs struggle to determine from the stream alone when finer temporal perception is needed. FastBench provides a testbed for high-dynamic streaming video understanding. Code and data: https://github.com/Ashone3/FastBench.

Problem

• Streaming VLMs
• High-dynamic perception
• Video understanding
• Benchmark
• Temporal sampling

Innovation

• Streaming VLMs
• High-dynamic perception
• FastBench
• ProactiveFrame
• Dual-tier sliding window