VoxMem: Benchmarking Multimodal Memory in Large Audio Language Models

2026-09-26

AI Summary

This study addresses the limitations of existing evaluation benchmarks for large audio language models, which predominantly focus on textual content within single-session scenarios while neglecting non-semantic acoustic information, thereby failing to comprehensively assess multimodal memory capabilities. To this end, we propose VoxMem, a benchmark introducing the first evaluation framework that jointly characterizes acoustic evidence types and memory operation dimensions. Comprising 34,743 multi-session speech instances, VoxMem incorporates a hierarchical context budget mechanism to systematically evaluate fifteen mainstream models across context windows ranging from 8K to 64K tokens. Our findings reveal significant deficiencies in current models: none achieves accuracy exceeding 40% under a 32K context window, and their retention of non-semantic cues, such as speaker identity, remains substantially inferior to that of textual content.

Abstract

Spoken conversational systems must recover information from prior interactions (i.e., memory), yet relevant information in speech extends beyond what was said to who said it, how it was spoken, and what was audible, information that exists only in the audio signal and cannot be recovered from a transcript. Beyond what to remember, memory also demands diverse operations: retrieving a single fact, integrating evidence across turns, tracking an evolving state. Real interactions further unfold across sessions, meaning information accumulates across distinct episodes rather than a single continuous recording. Existing benchmarks fall short on all three dimensions: they focus primarily on lexical content, adopt limited and ad hoc memory operations, and treat memory as a single-session problem. We argue that principled memory evaluation requires jointly characterizing the acoustic evidence to be retained and the operations applied to it, and introduce a taxonomy along these two axes. Building on this taxonomy, we present VoxMem: 3,196 evaluation instances over 34,743 spoken sessions (177 hours) crossing four acoustic evidence types (speech semantics, speaker identity, paralinguistic cues, environmental sound) with four memory operations (information extraction, multi-session reasoning, temporal tracking, and answer refusal), grounded in multi-session histories and stratified across context budgets from 8K to 64K tokens. Evaluating 15 LALMs, no model exceeds 40% at 32K. Models retain what was said far better than who said it, how, or what was audible, a gap that widens for complex operations, grows with history length, and manifests as qualitatively distinct failure modes across evidence types. VoxMem aims to provide a foundation to measure and drive progress on the full scope of spoken conversational memory.

Problem

• Multimodal Memory
• Large Audio Language Models
• Spoken Conversational Systems
• Benchmarking
• Multi-session Reasoning

Innovation

• Multimodal Memory Benchmark
• Large Audio Language Models
• Acoustic Evidence Taxonomy
• Multi-session Reasoning
• Spoken Conversational Systems