Seen, Said, or Forgotten? A Causal Audit of Visual KV Memory Across Dialog Turns

2026-07-28

AI Summary

This work addresses the lack of causal evaluation in existing visual dialogue systems regarding when to safely forget visual key-value (KV) memory during multi-turn interactions, a flaw that often leads to premature discarding of critical information due to neglecting dependencies in subsequent questions. The authors propose the Causal Visual Memory Audit (CVMA) framework, which leverages a paired single-pass prefill architecture to systematically quantify the impact of omitting specific image regions, entire images, or prior responses on the accuracy of future answers. Experiments reveal that current attention mechanisms rank useful visual regions even below random chance and demonstrate that forgetting is only safe when future visual dependency is low or relevant facts have already been explicitly stated. This study exposes fundamental limitations in prevailing KV compression strategies and establishes verifiable causal criteria for visual memory management.

Abstract

Stateful multimodal assistants encode an image once but may answer questions about it many turns later. Attention-guided visual-KV eviction assumes that evidence irrelevant now will remain dispensable, although future questions are unknown. We ask when a visual fact is actually safe to forget and introduce the Causal Visual Memory Audit (CVMA), a paired single-prefill framework that tests what later answers lose when a visual region, the whole image, or prior assistant text becomes unavailable. On VisDial and ConvBench, current attention can rank future-useful regions worse than random even though a diagnostic marginal-utility control shows substantial selection headroom. Aggregate scores hide this failure when later turns do not need vision; controlled and stock-generated histories reveal a second escape route, in which assistant-text KV replaces image KV for facts already stated but not reliably for unstated facts. In the tested stacks, safe forgetting is supported by low future visual dependence or fact-specific verbalization---not by low current attention.

Problem

• visual memory
• multimodal dialogue
• memory eviction
• attention mechanism
• causal audit

Innovation

• Causal Visual Memory Audit
• visual KV memory
• attention-guided eviction
• multimodal dialogue
• memory safety