Semantic-Aligned Structural Abstraction for Multimodal Sentiment Analysis

2026-07-30

AI Summary

This work addresses the limitation of current large language models in multimodal sentiment analysis, which struggle to capture deep emotional semantics arising from structural dynamics and contextual interactions, often relying only on superficial features. To overcome this, the authors propose SentiLLM, a novel framework that first converts continuous non-verbal signals into compact, semantically grounded text-like tokens through structure-aware semantic abstraction. It then introduces a dual-stream salience–context calibration mechanism that decouples focal and ambient streams, leveraging textual priors to guide the detection of sentiment shifts and achieve cross-modal semantic alignment. Finally, a lightweight, plug-and-play adapter module is integrated for efficient adaptation. Evaluated on MOSI, MOSEI, CH-SIMS, and CH-SIMS v2 benchmarks, SentiLLM achieves state-of-the-art performance with significantly improved discriminative capability while requiring only a minimal number of trainable parameters.

Abstract

Multimodal Sentiment Analysis (MSA) aims to interpret complex human emotions by integrating natural language with non-verbal modalities. Non-verbal modalities share a structural isomorphism with natural language, as both can be viewed as feature sequences evolving over time. This isomorphism enables the transformation of non-verbal modalities into text-like tokens for unified semantic reasoning. Large Language Models (LLMs), designed to understand and generate sequential data, can thus be utilized to interpret complex affective sequences. However, existing LLM-based methods primarily capture low-level superficial features, failing to model affective semantics arising from structural variations and contextual interactions. To address this limitation, we propose \textbf{SentiLLM}, a unified framework that leverages \textit{Semantic-Aligned Structural Abstraction} to distill continuous raw signals into compact, semantically meaningful tokens. Specifically, we introduce a \textit{Dual-Stream Salience-Context Calibration Mechanism}, which disentangles non-verbal feature sequences into a focus stream and an ambient stream. The focus stream captures salient sentiment shifts (e.g., facial expressions) guided by textual priors, while the ambient stream characterizes stable background states. Through calibrating these dynamic sentiment shifts against background states, SentiLLM effectively projects non-verbal modalities into a unified semantic space, making them naturally understandable for LLMs. Serving as a plug-and-play module, SentiLLM significantly improves discriminative performance with only a small number of trainable parameters. Our method achieves superior performance on four datasets, MOSI, MOSEI, CH-SIMS, and CH-SIMS v2, demonstrating the effectiveness of the structural abstraction paradigm in MSA. Our code is available at: \href{https://github.com/especiallyW/SentiLLM}.

Problem

• Multimodal Sentiment Analysis
• Large Language Models
• Affective Semantics
• Structural Abstraction
• Non-verbal Modalities

Innovation

• Semantic-Aligned Structural Abstraction
• Dual-Stream Salience-Context Calibration
• Multimodal Sentiment Analysis
• Large Language Models
• Non-verbal Modality Tokenization