Memento: Augmenting Personalized Memory via Practical Multimodal Wearable Sensing in Visual Search and Wayfinding Navigation

2025-04-28

AI Summary

To address short-term information forgetting caused by limited working memory capacity, this paper proposes a multimodal wearable sensing-based memory augmentation method that delivers real-time, context-aware prompts during visual search and path navigation tasks. The approach integrates eye-tracking, accelerometer, gyroscope, and ambient light data to train a lightweight cognitive state detection model, coupled with a rule-driven, in-situ prompting mechanism that dynamically triggers personalized visual cues. This work represents the first systematic application of multimodal wearable sensing for working memory enhancement. Experimental results demonstrate a 20–23% improvement in route recall accuracy, a 46% reduction in cognitive load and debriefing time, and an end-to-end latency of only 3.86 seconds—75% faster than conventional computer vision-based methods. The proposed framework effectively bridges critical gaps in low-latency, high-adaptivity short-term memory assistance technologies.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Planning, Routing, and Scheduling: Activity and Plan Recognition

Application Category

• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Working memory involves the temporary retention of information over short periods. It is a critical cognitive function that enables humans to perform various online processing tasks, such as dialing a phone number, recalling misplaced items' locations, or navigating through a store. However, inherent limitations in an individual's capacity to retain information often result in forgetting important details during such tasks. Although previous research has successfully utilized wearable and assistive technologies to enhance long-term memory functions (e.g., episodic memory), their application to supporting short-term recall in daily activities remains underexplored. To address this gap, we present Memento, a framework that uses multimodal wearable sensor data to detect significant changes in cognitive state and provide intelligent in situ cues to enhance recall. Through two user studies involving 15 and 25 participants in visual search navigation tasks, we demonstrate that participants receiving visual cues from Memento achieved significantly better route recall, improving approximately 20-23% compared to free recall. Furthermore, Memento reduced cognitive load and review time by 46% while also substantially reducing computation time (3.86 seconds vs. 15.35 seconds), offering an average of 75% effectiveness compared to computer vision-based cue selection approaches.

Problem

• Enhancing short-term memory in daily tasks using wearables
• Improving visual search and navigation via real-time cues
• Reducing cognitive load with multimodal sensor-based assistance

Innovation

• Multimodal wearable sensing for cognitive state detection
• Intelligent in situ visual cues for memory enhancement
• Reduced cognitive load with efficient computation time