Hybrid Deep Learning Model to Estimate Cognitive Effort from fNIRS Signals in Educational Game Playing

2025-04-03

arXiv.org

AI Summary

This study aims to enable real-time estimation of learners’ cognitive effort (CE) using functional near-infrared spectroscopy (fNIRS) ΔHbO signals and task performance scores, to support dynamic educational content adaptation and learning-state interventions. Method: We propose the first fNIRS-specific hybrid deep learning model combining convolutional neural networks (CNNs) and gated recurrent units (GRUs), jointly quantifying relative neural efficiency (RNE) and relative neural involvement (RNI). We further validate the effectiveness of transferring deep features to XGBoost—emphasizing CE trend modeling over point-wise prediction accuracy. Results: The CNN-GRU model achieves 73.08% test accuracy, with strong RNE/RNI trend alignment. Empirical analysis reveals that brief rest periods significantly alleviate high-CE states. Collectively, this work establishes an interpretable, intervention-capable neurocognitive closed loop for adaptive educational systems.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Neuro-Symbolic Learning
• Intelligent Robots: State Estimation

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

This study estimates cognitive effort (CE) based on functional near-infrared spectroscopy (fNIRS) data and performance scores using a hybrid deep learning model. The estimation of CE enables educators to modify material to enhance learning effectiveness and student engagement. Relative neural efficiency (RNE) and relative neural involvement (RNI) are two metrics that have been used to represent CE. To estimate RNE and RNI we need hemodynamic response in the brain and the performance score of a task.We collected oxygenated hemoglobin ($Delta mathrm{HbO}$). Sixteen participants answered 16 questions in a unity-based educational game, each with a 30-second response time. We used deep learning models to predict the performance score and estimate RNE and RNI to understand CE. The study compares traditional machine learning techniques with deep learning models such as CNN, LSTM, BiLSTM, and a hybrid CNN-GRU to determine which approach provides better accuracy in predicting performance scores. The result shows that the hybrid CNN-GRU gives better performance with 78.36% training accuracy and 73.08% test accuracy than other models. We performed XGBoost on the extracted GRU feature and got the highest accuracy (69.23%). This suggests that the features learned from this hybrid model generalize better even in traditional machine learning algorithms. We used the $Delta mathrm{HbO}$ and predicted score to calculate RNE and RNI to observe cognitive effort in our four test cases. Our result shows that even with moderate accuracy, the predicted RNE and RNI closely follows the actual trends. we also observed that when participants were in a state of high CE, introducing rest led decrease of CE. These findings can be helpful to design and improve learning environments and provide valuable insights in learning materials.

Problem

• Estimating cognitive effort from fNIRS signals using hybrid deep learning
• Comparing deep learning models to predict performance scores accurately
• Analyzing RNE and RNI to enhance learning environments and materials

Innovation

• Hybrid CNN-GRU model for cognitive effort estimation
• fNIRS data and performance scores integration
• XGBoost on GRU features for accuracy