Beyond Generative Decoding: Discriminative Hidden-State Readout from a Native Omni-Modal LLM for Multimodal Sentiment Analysis

2026-06-04

AI Summary

This work addresses the limitations of existing generative decoding approaches in multimodal sentiment analysis, which incur precision loss, uninterpretable outputs, and inference latency by forcing continuous sentiment scores into discrete text tokens. To overcome these issues, we introduce—for the first time—a discriminative readout mechanism within the Thinker module of the full-modality large language model Qwen2.5-Omni-7B. This mechanism directly predicts continuous sentiment values via a lightweight regression head applied to the hidden states of the last non-padding token, bypassing autoregressive generation entirely. Combined with 4-bit quantization and QLoRA fine-tuning, our approach achieves state-of-the-art performance on CMU-MOSI (MAE 0.551, Corr 0.888) and CMU-MOSEI (MAE 0.506, Corr 0.790) while training only 1.14% of the model parameters, significantly enhancing accuracy, efficiency, and stability—even enabling effective training on a single consumer-grade GPU.

Abstract

Multimodal sentiment analysis (MSA) infers human affect from language, acoustic, and visual signals. Recent methods increasingly adapt large multimodal models (LMMs) via generative readout: prompting the model to emit a sentiment score as a text string. While convenient, this ties continuous regression to discrete autoregressive decoding, incurring unmeasured costs. We revisit this readout mechanism and propose a discriminative formulation built on the Thinker module of a native omni-modal LLM (Qwen2.5-Omni-7B). Instead of text decoding, we map the final-layer hidden state of the last non-padding token to a continuous score via a lightweight regression head in a single forward pass. Using 4-bit quantization and low-rank adaptation (QLoRA), the entire 7B pipeline -- including video and audio processing -- trains on a single consumer GPU (RTX 5090, 32 GB) with 10-21 GB peak memory and 1.14% trainable parameters. Through a controlled comparison fixing the backbone, data, and LoRA configuration, we isolate the impact of the readout. On CMU-MOSI and CMU-MOSEI, our discriminative readout reaches state-of-the-art accuracy without task-specific feature engineering (MOSI: MAE 0.551, Corr 0.888; MOSEI: MAE 0.506, Corr 0.790) and exhibits strong multi-seed stability. In contrast, the generative readout -- even after equivalent supervised training -- more than doubles the mean absolute error, yields unparsable or out-of-range outputs (2.8% zero-shot), and suffers from higher latency. Modality ablations reveal a text-dominant regime on CMU-MOSI. Our findings indicate that how an LMM is read out is as consequential as how it is trained, demonstrating that a discriminative readout offers a more accurate, efficient, and reliable alternative for continuous MSA.

Problem

• multimodal sentiment analysis
• generative decoding
• discriminative readout
• large multimodal models
• continuous regression

Innovation

• discriminative readout
• omni-modal LLM
• multimodal sentiment analysis
• QLoRA
• hidden-state regression