ProMemAssist: Exploring Timely Proactive Assistance Through Working Memory Modeling in Multi-Modal Wearable Devices

2025-07-28

AI Summary

Wearable AI systems often rely on user-initiated triggers or predefined task logic, neglecting real-time cognitive states—leading to ill-timed interventions. Method: This paper proposes a proactive assistance framework grounded in dynamic visual working memory (WM) modeling. Leveraging multimodal wearable sensors—including smart glasses—it captures physiological and behavioral signals in real time; integrates cognitive theory to construct an encoding-interference WM representation model for fine-grained perception of cognitive load and attentional state; and introduces, for the first time in wearables, a WM-informed interruption cost–benefit trade-off framework to optimize intervention timing. The system integrates sensor fusion, WM modeling, real-time attention recognition, and LLM-augmented collaborative reasoning. Contribution/Results: A 12-participant user study demonstrates significant improvements over a pure-LLM baseline: +32.7% accuracy in assistance timing (p < 0.01) and significantly higher user engagement (p < 0.01), validating the efficacy and feasibility of cognition-aligned proactive intervention.

Technology Category

• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Humans and AI: Intelligent User Interfaces
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Wearable AI systems aim to provide timely assistance in daily life, but existing approaches often rely on user initiation or predefined task knowledge, neglecting users' current mental states. We introduce ProMemAssist, a smart glasses system that models a user's working memory (WM) in real-time using multi-modal sensor signals. Grounded in cognitive theories of WM, our system represents perceived information as memory items and episodes with encoding mechanisms, such as displacement and interference. This WM model informs a timing predictor that balances the value of assistance with the cost of interruption. In a user study with 12 participants completing cognitively demanding tasks, ProMemAssist delivered more selective assistance and received higher engagement compared to an LLM baseline system. Qualitative feedback highlights the benefits of WM modeling for nuanced, context-sensitive support, offering design implications for more attentive and user-aware proactive agents.

Problem

• Modeling working memory for proactive wearable assistance
• Balancing assistance value with interruption cost
• Enhancing context-aware support via multi-modal sensing

Innovation

• Real-time working memory modeling using multi-modal sensors
• Timing predictor balancing assistance value and interruption cost
• Context-sensitive support based on cognitive WM theories