Cognitive Prosthetic: An AI-Enabled Multimodal System for Episodic Recall in Knowledge Work

2026-03-02

AI Summary

This work addresses the detrimental impact of attention fragmentation and multimodal information overload on human episodic memory in modern knowledge work, a challenge inadequately met by existing tools that fail to integrate cognitive, physiological, and attentional context. To bridge this gap, we propose the first end-to-end, privacy-preserving, and modular AI-driven Contextual Prospective Memory System (CPMS). CPMS synchronously fuses speech transcripts, eye-tracking data, and physiological signals to generate locally processed, time-aligned, structured JSON memory records. It enables natural language–based retrieval along semantic, temporal, attentional, or physiological dimensions. Our system demonstrates, for the first time, the technical feasibility of transforming heterogeneous sensor streams into queryable episodic memories, validating a deployable, multimodal memory augmentation prototype in authentic workplace settings.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Cognitive Modeling & Cognitive Systems: Applications
• Humans and AI: Brain-Sensing and Analysis

Application Category

• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Security and Privacy: Data transparency and provenance

Abstract

Modern knowledge workplaces increasingly strain human episodic memory as individuals navigate fragmented attention, overlapping meetings, and multimodal information streams. Existing workplace tools provide partial support through note-taking or analytics but rarely integrate cognitive, physiological, and attentional context into retrievable memory representations. This paper presents the Cognitive Prosthetic Multimodal System (CPMS) --an AI-enabled proof-of-concept designed to support episodic recall in knowledge work through structured episodic capture and natural language retrieval. CPMS synchronizes speech transcripts, physiological signals, and gaze behavior into temporally aligned, JSON-based episodic records processed locally for privacy. Beyond data logging, the system includes a web-based retrieval interface that allows users to query past workplace experiences using natural language, referencing semantic content, time, attentional focus, or physiological state. We present CPMS as a functional proof-of-concept demonstrating the technical feasibility of transforming heterogeneous sensor data into queryable episodic memories. The system is designed to be modular, supporting operation with partial sensor configurations, and incorporates privacy safeguards for workplace deployment. This work contributes an end-to-end, privacy-aware architecture for AI-enabled memory augmentation in workplace settings.

Problem

• episodic memory
• knowledge work
• multimodal data
• cognitive prosthetic
• memory augmentation

Innovation

• episodic memory augmentation
• multimodal sensing
• natural language retrieval
• privacy-preserving AI
• cognitive prosthetic