Ambivalence/Hesitancy Recognition in Videos for Personalized Digital Health Interventions

2026-04-13

AI Summary

This study addresses the automatic recognition of ambivalence and hesitation exhibited by users during digital health interventions, aiming to enable personalized support. It pioneers the systematic integration of such emotion recognition into digital health contexts by proposing a multimodal deep learning framework that encompasses three paradigms: supervised learning, unsupervised domain adaptation, and zero-shot inference with large language models (LLMs). The work further introduces strategies for personalized domain adaptation and cross-modal conflict modeling. Experimental evaluation on the BAH dataset reveals the limited performance of existing approaches, underscoring the urgent need for improved spatiotemporal modeling and multimodal fusion mechanisms, thereby charting a clear direction for future research in this emerging area.

Technology Category

• Cognitive Modeling & Cognitive Systems: Affective Computing
• Humans and AI: Emotional Intelligence
• Machine Learning: Multimodal Learning

Application Category

• User Modeling, Personalization and Recommendation: Studies of user behavior, including longitudinal effects of personalized systems
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Using behavioural science, health interventions focus on behaviour change by providing a framework to help patients acquire and maintain healthy habits that improve medical outcomes. In-person interventions are costly and difficult to scale, especially in resource-limited regions. Digital health interventions offer a cost-effective approach, potentially supporting independent living and self-management. Automating such interventions, especially through machine learning, has gained considerable attention recently. Ambivalence and hesitancy (A/H) play a primary role for individuals to delay, avoid, or abandon health interventions. A/H are subtle and conflicting emotions that place a person in a state between positive and negative evaluations of a behaviour, or between acceptance and refusal to engage in it. They manifest as affective inconsistency across modalities or within a modality, such as language, facial, vocal expressions, and body language. While experts can be trained to recognize A/H, integrating them into digital health interventions is costly and less effective. Automatic A/H recognition is therefore critical for the personalization and cost-effectiveness of digital health interventions. Here, we explore the application of deep learning models for A/H recognition in videos, a multi-modal task by nature. In particular, this paper covers three learning setups: supervised learning, unsupervised domain adaptation for personalization, and zero-shot inference via large language models (LLMs). Our experiments are conducted on the unique and recently published BAH video dataset for A/H recognition. Our results show limited performance, suggesting that more adapted multi-modal models are required for accurate A/H recognition. Better methods for modeling spatio-temporal and multimodal fusion are necessary to leverage conflicts within/across modalities.

Problem

• ambivalence
• hesitancy
• digital health interventions
• multimodal recognition
• personalization

Innovation

• ambivalence/hesitancy recognition
• multimodal deep learning
• unsupervised domain adaptation
• zero-shot inference
• digital health interventions