H2HMem: A Multimodal Memory Benchmark for Agents in Human-Human Interactions

2026-06-08

AI Summary

Existing memory evaluation benchmarks are limited to single-user, text-only human–AI interactions, making them inadequate for assessing agents’ memory capabilities in multi-participant, multimodal human conversations. This work proposes the first memory evaluation framework tailored to human–human multimodal dialogues, introducing a novel dataset that encompasses both dyadic and multi-party scenarios. The authors establish a comprehensive evaluation protocol along three dimensions: memory recall, reasoning, and application. Experimental results using large language model agents reveal significant deficiencies in current systems regarding cross-modal, cross-participant, and cross-session memory construction and utilization. By addressing this critical gap in evaluation methodology, the study provides a foundational benchmark and clear directions for advancing memory capabilities in next-generation conversational agents.

Abstract

Large language model agents are increasingly deployed in human-human interaction settings, such as meeting assistants and clinical documentation systems, where they must observe conversations and retain information for downstream queries. Unlike traditional human-assistant settings, these environments are inherently multimodal, involve complex discourse phenomena such as anaphora and deixis, and contain asynchronous or conflicting information from multiple participants. However, existing memory benchmarks largely focus on single-user, text-only interactions, failing to capture these challenges. To address this gap, we introduce H2HMem, a Human-to-Human Multimodal Memory Benchmark for evaluating memory capabilities in complex human-human interactions. H2HMem includes both dyadic and multi-party conversations with multimodal information streams, and evaluates agents along three dimensions: memory recall, reasoning, and application. Experiments with advanced agents reveal substantial limitations in constructing, retaining, and utilizing memories across modalities, participants, and sessions, highlighting substantial room for improvement in next-generation LLM agents.

Problem

• multimodal memory
• human-human interaction
• anaphora
• deixis
• memory benchmark

Innovation

• multimodal memory
• human-human interaction
• memory benchmark
• LLM agents
• anaphora and deixis