V-Mem: Modality-Routed Retrieval for Long-Term Multimodal Agentic Memory

2026-08-02

AI Summary

This work addresses the challenge of effectively retrieving cross-modal evidence for visually grounded questions in multimodal dialogues where text and images are interwoven. The authors propose a modality-aware routing retrieval mechanism that first explicitly identifies the target modality of a query to select an appropriate retrieval pathway. To bridge the gap between semantic similarity and task relevance, they introduce LLM-generated hypothetical captions and image keywords as semantic anchors. Their approach further integrates a turn-aligned memory structure with an intra-modality matching strategy within a joint embedding space. Experimental results demonstrate significant improvements over existing baselines, achieving an LLM-judge score of 0.82 on Mem-Gallery (0.87 for image-related questions) and 0.69 on LoCoMo.

Abstract

Interaction between users and LLM agents is increasingly multimodal: conversations interleave text with images, and a later question may target either. Yet most agent memories are designed around text, and even the few that support multimodal conversations still fail on vision-related questions. We trace this failure to an assumption behind the similarity search they rely on: in the index space, a query lies close to the relevant evidence that answers it. In multimodal settings, two gaps break it. By the modality gap, a query lies closer to memory content of its own modality than to evidence in another, even in a trained joint embedding space. By the similarity-relevance gap, the content most similar to a query is often not the evidence that answers it, most acutely when a query carries both text and image and its evidence resembles neither part alone. We present V-Mem, a multimodal agentic memory system that routes retrieval by the modality of the query and that of the target evidence, both recognized from the query alone. To cross the modality gap, V-Mem organizes the conversation into rounds and returns the target-modality content from the same round as the match, without comparing across modalities. To close the similarity-relevance gap, it searches with an LLM-generated anchor that sits closer to the relevant evidence than the query does: a hypothetical caption for a text-only query seeking an image, and an enriched search anchor, the query text plus relevant keywords extracted from the query image, when the evidence is reachable only by combining the two. On Mem-Gallery, V-Mem reaches an LLM-judge score of 0.82 versus 0.56 for the second best, with the largest margin on questions carrying an image (0.87, no baseline above 0.47); on LoCoMo it scores 0.69 versus 0.58.

Problem

• multimodal memory
• modality gap
• similarity-relevance gap
• vision-related questions
• agent memory

Innovation

• modality-routed retrieval
• multimodal memory
• similarity-relevance gap
• LLM-generated anchor
• agentic memory