MemoCare: An Interactive Multimodal Mobile System for Automated Cognitive Screening

2026-10-07

AI Summary

This study addresses the lack of multimodal mobile cognitive screening systems by developing an automated English-Vietnamese bilingual screening platform using React Native, integrating voice, spatial, touch, and drawing tasks. We propose a novel multimodal automated scoring architecture that combines deterministic natural language processing rules with a ShuffleNetV2 convolutional neural network consensus mechanism to achieve standardized cross-lingual assessment. Experimental results demonstrate that all software tests were successfully passed, the drawing module attained a balanced accuracy of 91.33%, and clinical experts awarded a comprehensive rating of 4 out of 5. These findings validate the effectiveness and clinical applicability of the proposed system for cross-lingual cognitive screening.

Abstract

MemoCare is an interactive mobile system for automated multimodal cognitive screening. A React Native application combines spoken responses, temporal and spatial orientation, touchscreen actions, and visuoconstruction in complete English and Vietnamese workflows. Speech is transcribed by Google Speech-to-Text and scored locally with deterministic task-specific natural language processing rules; GPS coordinates are resolved by the MemoCare spatial module before answer matching; touch tasks are scored from interaction events; and the drawing task uses a three-model convolutional neural network consensus with separate visual interpretation. Software tests pass 151/151 predefined cases across speech/language, spatial-answer, and touch-interaction scoring, while spatial regression passes 48/48 four-country coordinate-resolution cases. For the drawing module, validation-selected ShuffleNetV2 x1.5 achieved 91.33% mean balanced accuracy and 78.87% exact three-criterion accuracy on a locked 71-image test set. Four clinician co-authors additionally inspected the end-to-end workflow, yielding a pooled median rating of 4/5 across eight criteria, with item-level medians ranging from 3 to 4.5. At MMM, attendees can directly try a shortened multimodal screening workflow and inspect automatic item-level and total scoring.

Problem

• cognitive screening
• multimodal assessment
• mobile health
• automated scoring

Innovation

• Multimodal cognitive screening
• Mobile health system
• Convolutional neural network consensus
• Natural language processing
• Automated scoring