Joint Modeling for Learning Decision-Making Dynamics in Behavioral Experiments

2025-06-03

AI Summary

Major depressive disorder (MDD) is associated with aberrant reward-based decision-making, yet existing computational models lack the capacity to dissociate distinct cognitive states underlying such impairments. Method: We propose a novel integrative framework combining reinforcement learning (RL), the drift-diffusion model (DDM), and hidden Markov modeling (HMM), featuring a “engagement–disengagement” bistable switching mechanism: the engagement state jointly models reward learning and evidence accumulation, while the disengagement state captures stochastic responding via a simplified DDM. Parameters are estimated using a generalized expectation-maximization algorithm. Results: The model reveals reduced overall engagement and slower decision dynamics specifically within the engagement state in MDD patients. In the EMBARC dataset, it significantly outperforms conventional single-state models. Critically, robust brain–behavior associations—linking neural activity to decision parameters—were detected exclusively in the engagement state, enabling state-specific neurocognitive decoding.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Intelligent Robots: State Estimation
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• User Modeling, Personalization and Recommendation: Psychology-informed user models and recommender systems
• Economics, Online Markets and Human Computation: Social networks and social learning
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Major depressive disorder (MDD), a leading cause of disability and mortality, is associated with reward-processing abnormalities and concentration issues. Motivated by the probabilistic reward task from the Establishing Moderators and Biosignatures of Antidepressant Response in Clinical Care (EMBARC) study, we propose a novel framework that integrates the reinforcement learning (RL) model and drift-diffusion model (DDM) to jointly analyze reward-based decision-making with response times. To account for emerging evidence suggesting that decision-making may alternate between multiple interleaved strategies, we model latent state switching using a hidden Markov model (HMM). In the ''engaged'' state, decisions follow an RL-DDM, simultaneously capturing reward processing, decision dynamics, and temporal structure. In contrast, in the ''lapsed'' state, decision-making is modeled using a simplified DDM, where specific parameters are fixed to approximate random guessing with equal probability. The proposed method is implemented using a computationally efficient generalized expectation-maximization algorithm with forward-backward procedures. Through extensive numerical studies, we demonstrate that our proposed method outperforms competing approaches under various reward-generating distributions, both with and without strategy switching. When applied to the EMBARC study, our framework reveals that MDD patients exhibit lower overall engagement than healthy controls and experience longer decision times when they do engage. Additionally, we show that neuroimaging measures of brain activities are associated with decision-making characteristics in the ''engaged'' state but not in the ''lapsed'' state, providing evidence of brain-behavioral association specific to the ''engaged'' state.

Problem

• Modeling decision-making dynamics in MDD patients using RL-DDM integration
• Detecting latent state switching in decision-making via HMM
• Linking neuroimaging measures to engaged-state decision characteristics

Innovation

• Integrates RL and DDM for decision analysis
• Uses HMM for latent state switching
• Employs efficient EM algorithm for implementation