Representation Decomposition for Learning Similarity and Contrastness Across Modalities for Affective Computing

2025-06-08

AI Summary

Existing methods struggle to jointly model the similarity and conflict between affective signals across modalities (e.g., image and text). To address this, we propose the first large language model (LLM)-based representation decomposition framework that explicitly disentangles cross-modal shared affective components from modality-specific cues. Our approach employs an attention-driven dynamic soft prompting mechanism for adaptive multimodal fusion and guides the fine-tuning of multimodal LLMs. The architecture integrates a pretrained multimodal encoder, a representation decomposition network, and a soft prompt generation module. Evaluated on three benchmark tasks—sentiment analysis, emotion recognition, and hate meme detection—our method consistently outperforms state-of-the-art approaches, achieving average accuracy gains of 2.3–4.1%. Notably, it is the first to jointly model both congruent and adversarial evidence in multimodal affective understanding.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

Multi-modal affective computing aims to automatically recognize and interpret human attitudes from diverse data sources such as images and text, thereby enhancing human-computer interaction and emotion understanding. Existing approaches typically rely on unimodal analysis or straightforward fusion of cross-modal information that fail to capture complex and conflicting evidence presented across different modalities. In this paper, we propose a novel LLM-based approach for affective computing that explicitly deconstructs visual and textual representations into shared (modality-invariant) and modality-specific components. Specifically, our approach firstly encodes and aligns input modalities using pre-trained multi-modal encoders, then employs a representation decomposition framework to separate common emotional content from unique cues, and finally integrates these decomposed signals via an attention mechanism to form a dynamic soft prompt for a multi-modal LLM. Extensive experiments on three representative tasks for affective computing, namely, multi-modal aspect-based sentiment analysis, multi-modal emotion analysis, and hateful meme detection, demonstrate the effectiveness of our approach, which consistently outperforms strong baselines and state-of-the-art models.

Problem

• Decomposing multi-modal representations into shared and unique components
• Improving emotion recognition from conflicting visual and textual cues
• Enhancing affective computing via dynamic LLM prompts for cross-modal fusion

Innovation

• Deconstructs representations into shared and specific components
• Uses pre-trained encoders for modality alignment
• Integrates signals via attention for dynamic LLM prompts