LLaSA: A Multimodal LLM for Human Activity Analysis Through Wearable and Smartphone Sensors

2024-06-20

AI Summary

While existing wearable devices collect rich inertial measurement unit (IMU) motion data, mainstream methods only support behavioral classification and cannot address causal or semantically grounded natural language questions—e.g., “Why did this occur?” or “What does it signify?” Method: We propose a sensor-driven causal question-answering paradigm and introduce LLaSA, the first multimodal large language model (13B parameters) designed specifically for raw IMU sequences. LLaSA enables open-domain behavioral causal explanation and context-aware reasoning via a time-series encoder integrated with instruction tuning, scientific accuracy constraints, and sensor–text alignment training. Contribution/Results: We establish three novel benchmarks—SensorCaps, OpenSQA, and Tune-OpenSQA—to formalize sensor–language joint modeling. Experiments demonstrate that LLaSA significantly outperforms commercial LLMs on both public and real-world benchmarks, generating answers with strong interpretability and causal consistency. Code and datasets are publicly released.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Question Answering
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Wearables generate rich motion data, yet current systems only classify what happened - failing to support natural questions about why it happened or what it means. We introduce LLaSA (Large Language and Sensor Assistant), a compact 13B model that enables ask-anything, open-ended question answering grounded in raw IMU data. LLaSA supports conversational, context-aware reasoning - explaining the causes of sensor-detected behaviors and answering free-form questions in real-world scenarios. It is tuned for scientific accuracy, coherence, and response reliability. To advance this new task of sensor-based QA, we release three large-scale datasets: SensorCaps, OpenSQA, and Tune-OpenSQA. Together, these resources define a new benchmark for sensor-language models. LLaSA consistently produces interpretable, causal answers and outperforms commercial LLMs across both public and real-world settings. Our code repository and datasets can be found at https://github.com/BASHLab/LLaSA.

Problem

• Enables open-ended QA for sensor data analysis
• Explains causes of behaviors detected by sensors
• Advances sensor-based QA with new datasets

Innovation

• Compact 13B model for sensor-based QA
• Open-ended question answering with IMU data
• Context-aware reasoning for behavior explanation