Style over Substance: A Shortcut Audit of Emotion-Description Preference Evaluation

2026-07-20

AI Summary

This work addresses a critical flaw in current multimodal sentiment understanding evaluation: human preferences for generated descriptions are often swayed by superficial stylistic cues—such as generation style—rather than genuine comprehension of video content. To systematically audit shortcut biases in the EmoPrefer benchmark, the authors propose a blind probing model that operates without access to textual, audio, or visual content. Through logistic regression, generator identification, length-matched subset analysis, and an ODIN-inspired disentanglement diagnostic, they demonstrate that description length and generator identity alone can predict human preferences with remarkable efficacy. The blind model achieves 65.8 WAF on EmoPrefer-V2, nearly matching the 66.8 WAF of fine-tuned models; generator identification accuracy reaches 99.5%; and controlling for length eliminates significant gains from multimodal inputs. This study is the first to expose prominent stylistic shortcuts in affective description preference evaluation, establishes a strong content-agnostic baseline, and calls for more rigorous evaluation protocols.

Abstract

Preference over model-generated emotion descriptions is emerging as a standard evaluation metric for multimodal emotion understanding, exemplified by the MER2026 MER-Prefer track on EmoPrefer. Such benchmarks assume that predicting the preferred description requires grounded cross-modal understanding of the video. We conduct a systematic shortcut audit of EmoPrefer using content-blind probes. A simple logistic regression using only description length and generator identity, without processing the text, video, or audio, performs comparably to LoRA-finetuned 7B text and audio-visual judges (65.8 versus 66.8 WAF on EmoPrefer-V2). Generator identity is recoverable from description text with 99.5 percent accuracy, every candidate pair contrasts two distinct generators, and the human preference labels agree with a fold-exclusive per-generator win-rate prior on 66 percent of the evaluated pairs. When the human label conflicts with this prior, trained judges still follow the style prior on 63 to 80 percent of the pairs. On a length-matched subset that neutralizes verbosity bias, the tested media configurations yield no statistically significant improvement, while an ODIN-inspired diagnostic that decouples the style shortcut leaves its content head near chance. These results do not imply that human preferences are inherently stylistic or that the descriptions contain no emotional information. Instead, they show that the current scores can be reached without verifying either description against the video. We recommend source-balanced pairing, strict length control, counter-stereotypical sliced reporting, and multi-annotator consensus for future cross-generator evaluations. Code is available at https://github.com/jiabingyang01/EmoPrefer-Audit.

Problem

• emotion understanding
• preference evaluation
• shortcut learning
• multimodal benchmarking
• style bias

Innovation

• shortcut audit
• emotion description preference
• style bias
• content-blind probe
• multimodal evaluation