A Heterogeneous Multimodal Graph Learning Framework for Recognizing User Emotions in Social Networks

2025-01-13

AI Summary

Personalized emotion recognition in social networks remains challenging due to the complex, heterogeneous, and dynamic nature of user–content–interaction relationships and multimodal signals. Method: This paper proposes HMG-Emo, a heterogeneous multimodal graph learning framework that models users, posts, and interactions as heterogeneous nodes, and encodes textual, visual, and behavioral modalities as dynamic edges to construct a dynamic heterogeneous graph. It is the first to jointly integrate deep multimodal encoders (BERT/ResNet) with heterogeneous graph neural networks (HGNNs), augmented by a novel dynamic contextual gating fusion module for adaptive cross-modal weighting and end-to-end joint optimization. Results: Extensive experiments on multiple real-world social media datasets demonstrate that HMG-Emo significantly outperforms handcrafted-feature-based state-of-the-art methods, achieving an average accuracy improvement of 8.3%. These results validate the effectiveness and advancement of heterogeneous graph structural modeling and dynamic multimodal fusion for personalized emotion prediction.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: Social recommender systems and personalization for the social Web
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

The rapid expansion of social media platforms has provided unprecedented access to massive amounts of multimodal user-generated content. Comprehending user emotions can provide valuable insights for improving communication and understanding of human behaviors. Despite significant advancements in Affective Computing, the diverse factors influencing user emotions in social networks remain relatively understudied. Moreover, there is a notable lack of deep learning-based methods for predicting user emotions in social networks, which could be addressed by leveraging the extensive multimodal data available. This work presents a novel formulation of personalized emotion prediction in social networks based on heterogeneous graph learning. Building upon this formulation, we design HMG-Emo, a Heterogeneous Multimodal Graph Learning Framework that utilizes deep learning-based features for user emotion recognition. Additionally, we include a dynamic context fusion module in HMG-Emo that is capable of adaptively integrating the different modalities in social media data. Through extensive experiments, we demonstrate the effectiveness of HMG-Emo and verify the superiority of adopting a graph neural network-based approach, which outperforms existing baselines that use rich hand-crafted features. To the best of our knowledge, HMG-Emo is the first multimodal and deep-learning-based approach to predict personalized emotions within online social networks. Our work highlights the significance of exploiting advanced deep learning techniques for less-explored problems in Affective Computing.

Problem

• Multimodal Data
• Emotion Recognition
• Social Networks

Innovation

• Multi-modal Deep Learning
• Personalized Sentiment Prediction
• Social Media Data Fusion