Beyond Vision: How Large Language Models Interpret Facial Expressions from Valence-Arousal Values

2025-02-08

AI Summary

This work investigates whether large language models (LLMs) can perform affective semantic understanding solely from valence-arousal (VA) numerical representations—without visual input. We evaluate zero-shot emotion classification and prompt-driven semantic description generation on the IIMI (basic emotions) and Emotic (complex emotions) datasets; VA features are extracted by FaceChannel, and models including GPT and Llama are assessed. Results show that LLMs exhibit limited performance on VA-based classification—particularly in distinguishing non-binary emotions—yet achieve strong performance in semantic description generation (BLEU-4 = 0.62; human-rated semantic similarity = 4.3/5), demonstrating robust mapping from abstract affective dimensions to natural language. To our knowledge, this is the first study to empirically validate that LLMs can directly interpret non-visual, structured affective representations. The findings establish a novel paradigm for lightweight, bias-mitigated affective computing grounded in interpretable, modality-agnostic emotional features.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Affective Computing
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Large Language Models primarily operate through text-based inputs and outputs, yet human emotion is communicated through both verbal and non-verbal cues, including facial expressions. While Vision-Language Models analyze facial expressions from images, they are resource-intensive and may depend more on linguistic priors than visual understanding. To address this, this study investigates whether LLMs can infer affective meaning from dimensions of facial expressions-Valence and Arousal values, structured numerical representations, rather than using raw visual input. VA values were extracted using Facechannel from images of facial expressions and provided to LLMs in two tasks: (1) categorizing facial expressions into basic (on the IIMI dataset) and complex emotions (on the Emotic dataset) and (2) generating semantic descriptions of facial expressions (on the Emotic dataset). Results from the categorization task indicate that LLMs struggle to classify VA values into discrete emotion categories, particularly for emotions beyond basic polarities (e.g., happiness, sadness). However, in the semantic description task, LLMs produced textual descriptions that align closely with human-generated interpretations, demonstrating a stronger capacity for free text affective inference of facial expressions.

Problem

• LLMs infer facial expressions from Valence-Arousal values.
• LLMs classify facial expressions into basic and complex emotions.
• LLMs generate semantic descriptions of facial expressions.

Innovation

• LLMs interpret facial Valence-Arousal values
• VA values replace raw visual inputs
• LLMs excel in semantic description tasks