SuperMemory-VQA: An Egocentric Visual Question-Answering Benchmark for Long-Horizon Memory

2026-05-30

AI Summary

This work addresses the lack of effective modeling of human memory demands in long-term, first-person video and the absence of suitable evaluation benchmarks. To bridge this gap, the authors introduce a real-world, long-horizon visual question answering (VQA) dataset grounded in egocentric multimodal signals—including RGB video, audio transcripts, eye gaze, IMU data, and SLAM trajectories—collected via AI glasses. The dataset comprises 4,853 human-verified, grounded question-answer pairs covering objects, locations, intentions, scenes, timelines, and conversational memory, and uniquely incorporates an “unanswerable” option to assess model robustness against hallucination. Experimental results demonstrate that current state-of-the-art agents and large language models perform poorly on this task, confirming the benchmark’s authenticity and difficulty, and highlighting a clear path toward developing reliable embodied memory systems.

Abstract

AI glasses present a compelling platform for AI agents to serve as personalized memory assistants. To be genuinely useful, such systems must move beyond short-term video comprehension and address memory gaps that humans experience for practical, personal, or social purposes over longitudinal egocentric video streams. However, existing egocentric datasets predominantly focus on action recognition or generic QAs from short clips, measuring perceptual capabilities rather than realistic human memory needs. We introduce SuperMemory-VQA, an egocentric visual question answering (VQA) dataset for evaluating AI assistants on practical, long-horizon memory tasks. It contains 52.9 hours of everyday activities recorded with AI glasses, including synchronized RGB video, audio transcription, eye gaze, IMU, and SLAM trajectories. Through a human-verified annotation pipeline, we construct grounded 4,853 question-answer pairs that span object and location memory, intent recall, visual scene recall, timeline reconstruction, conversational memory, and in-context retrieval. Each question is posed as multiple-choice with an explicit "unanswerable" option to test hallucination robustness. Benchmarking leading agentic frameworks and LLM backbones reveals that existing systems remain far from reliable on real-world memory tasks, highlighting the need for new architectures for grounded AI memory that can answer only when evidence is sufficient. A participant survey further supports that our questions are realistic, useful, and aligned with everyday memory needs.

Problem

• egocentric vision
• visual question answering
• long-horizon memory
• memory assistance
• AI glasses

Innovation

• egocentric vision
• long-horizon memory
• visual question answering
• multimodal grounding
• hallucination robustness