From 'What' to 'How' and 'Why': Sharing LLM-Generated Retrospective Summaries of Older Adults' Passive Tracking Data with Remote Family Members

2026-06-02

AI Summary

This study addresses the challenge of transforming multimodal passive sensing data from older adults into comprehensible and meaningful retrospective summaries for remote family members to support caregiving decisions and emotional connection. To this end, the authors propose a semantic transition framework that progresses from “What” to “How” and “Why,” leveraging a large language model (LLM)-based, multi-layered multi-agent narrative summarization mechanism that integrates objective behavioral data with contextual awareness to generate explanatory insights. Through iterative refinement via technology probes and user-centered design, the approach was evaluated by 11 remote family members and demonstrated significantly higher ratings than baseline methods in satisfaction, perceived usefulness, trustworthiness, and willingness to adopt, thereby enhancing understanding and acceptance of remote caregiving practices.

Abstract

With the growing prevalence of modern ubiquitous computing technologies, multi-modal tracking systems hold promise for providing timely awareness and reassurance to stakeholders such as remote family members (RFMs) of older adults, who play a central role in care coordination. However, combining heterogeneous data streams into high-level, meaningful content - such as retrospective summaries - remains challenging. While recent work has demonstrated the promise of large language models (LLMs) for interpreting multi-modal tracking data, less attention has been given to generating narrative accounts for stakeholders like RFMs, who possess rich personal knowledge of older adults and strong emotional responsibility, yet have limited visibility into their daily lives and limited capacity for caregiving. In this work, we explore how LLMs can be used to generate retrospective summaries from multi-modal tracking data for RFMs of older adults. We leveraged and customized an existing system, Vital Insight, to generate initial summaries on different dates and data availability scenarios as technology probes, and conducted interviews with 11 RFMs to gather feedback. Based on these insights, we redesigned the system into a multi-layer, multi-agent, insight-driven summary approach that builds from objective statistics and descriptions to enriched, context-aware narratives. We then compared the redesigned summaries with the initial versions through a survey with the same 11 RFMs and found significant improvements in satisfaction, perceived helpfulness, trust, and willingness to receive the summaries. We conclude by presenting design implications for AI-generated summaries for RFMs and broader contexts, emphasizing the need to support RFMs' sensemaking shift from simply presenting ''What'' data were collected, to explaining ''How'' is my loved one doing and ''Why''.

Problem

• passive tracking data
• retrospective summaries
• remote family members
• older adults
• sensemaking

Innovation

• large language models
• multi-modal tracking data
• retrospective summarization
• context-aware narratives
• remote family caregivers