A Modular Workflow for Multimodal Reading Experiments

2026-08-06

AI Summary

This study addresses the lack of ecologically valid multimodal frameworks for online reading experiments. It proposes a web-based, modular experimental platform that enables real-time synchronous acquisition of eye-tracking, EEG, and mouse/keyboard interaction data, with precise temporal alignment of multisource signals achieved via Lab Streaming Layer. By integrating browser-based eye tracking and fixation-triggered EEG epoching, the framework dynamically links multimodal data to textual content at the word, sentence, and region-of-interest levels. Designed for reusability and extensibility, the platform supports flexible integration of multiple sensors and has been successfully applied in selective exposure research, enabling real-time gaze metric computation, immediate neural data processing, and dynamic within-session text annotation—thereby advancing integrated analysis of behavioral, neural, and linguistic markers during naturalistic reading.

Abstract

We introduce a web-based modular workflow for real-time multimodal experiments in naturalistic online reading. The workflow integrates eye tracking, EEG, and interaction data from mouse and keyboard, synchronizes them via Lab Streaming Layer, and links gaze to browser-based text at the word, sentence, and AOI levels. It is designed as a reusable experimental procedure that can be adapted to different sensors, tasks, and analysis goals. As a use case, we apply the workflow to a study of selective exposure in online news search and reading. During the experiment, gaze-derived measures are computed online, while EEG and other synchronized streams are processed immediately after task sessions based on fixation-triggered segmentation. The resulting behavioral, neural, and linguistic metrics support selecting text passages for targeted post-task rating or labelling within the same lab session. The workflow thus provides a general basis for multimodal research on reading and related cognitive processes, and supports the empirical validation, in ecological contexts, of constructs that are typically operationalised through self-report measures.

Problem

• multimodal reading
• eye tracking
• EEG
• naturalistic reading
• data synchronization

Innovation

• modular workflow
• multimodal synchronization
• gaze-text alignment
• real-time EEG processing
• ecological reading experiments