SenseSeek Dataset: Multimodal Sensing to Study Information Seeking Behaviors

2025-07-19

AI Summary

This study addresses the challenge of modeling cross-stage cognitive mechanisms in information-seeking behavior. We propose a multimodal sensing approach leveraging consumer-grade wearable devices to simultaneously capture electrodermal activity (EDA), electroencephalography (EEG), pupil diameter, eye-tracking trajectories, and limb motion data—integrated with screen recording, task responses, and extraction of 258 behavioral and physiological features. This yields the first fine-grained physiological-behavioral dataset covering the full information-seeking process: information need recognition, query formulation, query submission, and result evaluation—comprising 235 complete search trials from 20 participants. Through multimodal fusion modeling, we achieve significant improvements in stage-wise identification accuracy for cognitive intent and interaction patterns. The dataset and methodology establish a new benchmark and a reproducible technical framework for personalized modeling of information behavior.

Technology Category

• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Information processing tasks involve complex cognitive mechanisms that are shaped by various factors, including individual goals, prior experience, and system environments. Understanding such behaviors requires a sophisticated and personalized data capture of how one interacts with modern information systems (e.g., web search engines). Passive sensors, such as wearables, capturing physiological and behavioral data, have the potential to provide solutions in this context. This paper presents a novel dataset, SenseSeek, designed to evaluate the effectiveness of consumer-grade sensors in a complex information processing scenario: searching via systems (e.g., search engines), one of the common strategies users employ for information seeking. The SenseSeek dataset comprises data collected from 20 participants, 235 trials of the stimulated search process, 940 phases of stages in the search process, including the realization of Information Need (IN), Query Formulation (QF), Query Submission by Typing (QS-T) or Speaking (QS-S), and Relevance Judgment by Reading (RJ-R) or Listening (RJ-L). The data includes Electrodermal Activities (EDA), Electroencephalogram (EEG), PUPIL, GAZE, and MOTION data, which were captured using consumer-grade sensors. It also contains 258 features extracted from the sensor data, the gaze-annotated screen recordings, and task responses. We validate the usefulness of the dataset by providing baseline analysis on the impacts of different cognitive intents and interaction modalities on the sensor data, and effectiveness of the data in discriminating the search stages. To our knowledge, SenseSeek is the first dataset that characterizes the multiple stages involved in information seeking with physiological signals collected from multiple sensors. We hope this dataset can serve as a reference for future research on information-seeking behaviors.

Problem

• Study information seeking behaviors using multimodal sensing data
• Evaluate consumer-grade sensors in complex search scenarios
• Characterize multiple search stages with physiological signals

Innovation

• Uses consumer-grade sensors for data capture
• Integrates multimodal physiological and behavioral data
• Characterizes multiple stages of information seeking