Caved or Convinced: Temporal Sampling Gates Claim Deference in Video Large Language Models

2026-08-04

AI Summary

This study addresses a critical yet overlooked limitation in current video large language models (VLLMs): their tendency not only to uncritically accept erroneous temporal claims but also to incorrectly reject valid ones—a failure mode prior work has largely ignored. The authors distinguish between two root causes: unavailability of evidence due to unsampled frames and assertion-induced weighting bias. Through frame reordering and sampling-shift experiments, they pinpoint the origins of model failures and propose a novel inversion testing mechanism that calibrates model confidence without relying on user assertions. Integrated with a Youden index–based evaluation framework, this approach achieves accuracy of 0.92–1.00 on temporally discriminable events and prompts the model to abstain rather than guess randomly when event order is inherently indeterminable.

Abstract

When asked which of two events came first, video large language models can fail in two opposite ways: cave to a false claim, or reject a true one. Prior video sycophancy work measures only the first and mitigates it by teaching the model to trust the user less, a fix known in text and image models to worsen the second. In video, both failures come from two causes the literature treats as one: availability, whether the sparse sampled frames contain the two events, and weighting, whether that evidence is trusted over the user. We separate them with two interventions that keep the claim fixed: a frame-preserving reorder that flips the claim's truth, and a sampling-offset shift that captures or misses both events at a fixed frame budget. When the events are missed, the two twins present identical frames, so each of the nine models we evaluate accepts a true and a false claim at the same rate, making Youden's $J=0$ by construction. Availability is necessary but not sufficient. Five of the nine read the order, yet four of those five still cave to the false claim, so their deference hits a weighting ceiling. Since trust cannot be calibrated over evidence that was never sampled, we propose a reversal test that cancels the model's order prior by scoring the sampled frames forward and reversed, then answers, resamples, or abstains without reading the claim. The test raises the order accuracy to 0.92-1.00 on the models that read the order and abstains rather than guesses on those that cannot.

Problem

• video large language models
• temporal ordering
• claim deference
• frame sampling
• model sycophancy

Innovation

• temporal sampling
• video large language models
• claim deference
• reversal test
• evidence weighting