Persistent Object Narratives for Token-Efficient Video Language Models

2026-08-05

AI Summary

This work addresses the inefficiency of visual interface tokens in existing video large language models and their difficulty in explicitly modeling temporal associations among recurring objects. The authors propose SlotNarrative, a slot-based structured visual interface that parses videos into narrative sequences of persistent objects, representing each object and its temporal evolution through two compact token types: identity and state. SlotNarrative introduces a parameter-free memory mechanism that integrates multi-cue matching to cluster visual features into object slots, which are then fed into a frozen Video-LLM. Evaluated across multiple benchmarks, the method achieves superior accuracy and token efficiency using only 144 visual tokens, outperforming existing compact interfaces.

Abstract

Video large language models (Video-LLMs) have made strong progress in open-ended video understanding. However, their visual interfaces remain token-intensive and provide limited explicit structure for linking recurring object evidence across time. We introduce SlotNarrative, a slot-based interface that organizes a video into persistent object narratives represented by compact object-state tokens. Rather than compressing frame-wise features before establishing temporal correspondence, SlotNarrative first groups visual features into object-like slots and then associates recurring observations with clip-level object entries through a lightweight, parameter-free memory that integrates multiple complementary matching cues. Each retained entry is serialized into two token types: an identity token that summarizes persistent object appearance and a set of state tokens that encode segment-level appearance, geometry, visibility, and trajectory information. This design yields an interface of only 144 allocated visual-token positions for a frozen Video-LLM, independent of the number of sampled frames. Across multiple datasets, SlotNarrative achieves a favorable trade-off between accuracy and visual-token count compared with prior compact Video-LLM interfaces. Experimental results establish persistent object narratives as a compact, structured, and temporally organized visual interface for Video-LLMs. Our code will be made publicly available.

Problem

• Video-LLMs
• token efficiency
• object persistence
• temporal correspondence
• visual interface

Innovation

• SlotNarrative
• persistent object narratives
• token-efficient Video-LLMs
• object-centric representation
• temporal correspondence