IMAGINE: Adaptive Schema-Imagery Enhanced Composition for Composed Video Retrieval

2026-06-06

AI Summary

Existing compositional video retrieval methods struggle to model implicit semantic concepts in text that are not explicitly present in videos—such as inferring “birthday party” from “cake”—thereby limiting retrieval performance. To address this challenge, this work proposes IMAGINE, a novel framework that introduces a schematic imagery mechanism to concretely represent implicit semantics through dynamically generated multimodal prototypes. These prototypes adaptively modulate visual features, effectively bridging the semantic gap between explicit visual content and implicit user intent. The method is implemented as an end-to-end compositional retrieval network and achieves state-of-the-art performance on both video and image compositional retrieval tasks, significantly outperforming existing approaches across three mainstream benchmarks.

Abstract

Composed Video Retrieval (CVR) is designed to retrieve a target video that matches a reference video modified by a modification text. While existing methods explore cross-modal correspondences, they often assume modified objects appear directly in videos. However, modification texts frequently describe concepts not explicitly presented but implicitly expressed through semantically related visual cues (e.g., "cake" implying "birthday party"). Current approaches typically rely on aligning explicit feature representations within the concrete space, neglecting critical latent associations. To address this, we propose an adaptIve scheMa-ImAGery enhanced composItional NEtwork (IMAGINE). Unlike standard explicit matching, IMAGINE materializes implicit semantics (termed schema imagery) via dynamic multimodal prototypes. These prototypes capture shared latent concepts to adaptively modulate visual features, effectively injecting implicit guidance into the retrieval process. By bridging the gap between explicit visual contents and implicit retrieval intentions, IMAGINE achieves state-of-the-art performance in both CVR and Composed Image Retrieval (CIR) across three widely used benchmarks.

Problem

• Composed Video Retrieval
• implicit semantics
• schema imagery
• cross-modal correspondence
• latent associations

Innovation

• schema imagery
• composed video retrieval
• implicit semantics
• multimodal prototypes
• adaptive feature modulation