🤖 AI Summary
This work addresses the instability in multimodal sentiment analysis caused by representation misalignment when using independently pretrained modality encoders, which often fail to surpass strong text-only baselines. To overcome this limitation, the authors propose a multimodal framework centered on explicit alignment: visual inputs are first transformed into structured textual descriptions via a vision-language model, thereby unifying heterogeneous modalities into a shared linguistic space. Cross-modal alignment is further enhanced through semantic token selection and batch uniformity regularization, prioritizing alignment over complex fusion mechanisms. The proposed approach significantly outperforms existing unimodal and multimodal methods across multiple sentiment and emotion benchmarks, demonstrating the effectiveness and robustness of an alignment-first strategy.
📝 Abstract
Multimodal affective analysis aims to understand human sentiment and emotion by jointly modeling heterogeneous modalities such as text and images. However, multimodal models often fail to consistently outperform strong text-only baselines, with performance varying significantly across fusion strategies. In this work, we identify representation misalignment between independently pretrained modality encoders as a key bottleneck for effective multimodal learning, and show through controlled experiments that alignment prior to fusion is often more important than fusion complexity. To address this issue, we propose a unified multimodal affective analysis framework that leverages vision-language models (VLMs) to convert visual content into structured textual descriptions, projecting heterogeneous modalities into a shared linguistic space and enabling interpretable text-centric reasoning. To further improve robustness, we introduce a hybrid learning strategy that combines semantic token selection with a batch-level uniformity regularization objective, encouraging a more dispersed and stable global feature space while mitigating noise introduced by VLM-generated descriptions. Experiments on multiple multimodal sentiment and emotion benchmarks show that our method consistently outperforms strong unimodal and multimodal baselines, achieving state-of-the-art performance. Our analysis further highlights the critical role of representation alignment in multimodal affective learning.