MemPal: Leveraging Multimodal AI and LLMs for Voice-Activated Object Retrieval in Homes of Older Adults

2025-02-03

AI Summary

To address the challenge of object localization difficulties and increased caregiver burden arising from retrospective memory decline in older adults, this paper proposes MemPal, a wearable AI memory assistant. MemPal employs a first-person-view camera to capture real-time visual data and leverages a multimodal large language model (MLLM) to enable end-to-end, voice-driven object location retrieval. It supports natural-language queries, context-aware logging, lightweight edge inference, and proactive safety alerts. We introduce the first paradigm tailored for older adults integrating wearable vision, real-time situational modeling, and voice interaction—ensuring privacy-preserving, natural, and low-latency interaction (mean response time <1.2 seconds). Evaluated in real home environments with 15 older adults, MemPal significantly improved object retrieval success rates and achieved high user satisfaction, while supporting continuous multi-turn voice dialogue.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Older adults have increasing difficulty with retrospective memory, hindering their abilities to perform daily activities and posing stress on caregivers to ensure their wellbeing. Recent developments in Artificial Intelligence (AI) and large context-aware multimodal models offer an opportunity to create memory support systems that assist older adults with common issues like object finding. This paper discusses the development of an AI-based, wearable memory assistant, MemPal, that helps older adults with a common problem, finding lost objects at home, and presents results from tests of the system in older adults' own homes. Using visual context from a wearable camera, the multimodal LLM system creates a real-time automated text diary of the person's activities for memory support purposes, offering object retrieval assistance using a voice-based interface. The system is designed to support additional use cases like context-based proactive safety reminders and recall of past actions. We report on a quantitative and qualitative study with N=15 older adults within their own homes that showed improved performance of object finding with audio-based assistance compared to no aid and positive overall user perceptions on the designed system. We discuss further applications of MemPal's design as a multi-purpose memory aid and future design guidelines to adapt memory assistants to older adults' unique needs.

Problem

• Develops AI-based wearable memory assistant
• Assists older adults in finding lost objects
• Uses multimodal LLM for real-time memory support

Innovation

• Multimodal AI for memory support
• Voice-activated object retrieval system
• Wearable camera for visual context