Benchmarking Zero-Shot Facial Emotion Annotation with Large Language Models: A Multi-Class and Multi-Frame Approach in DailyLife

2025-02-18

AI Summary

This study investigates the feasibility and performance of large language models (LLMs) for zero-shot emotion recognition in real-life videos. Addressing the FERV39k-DailyLife dataset, it performs both fine-grained (seven classes: Angry, Disgust, Fear, Happy, Neutral, Sad, Surprise) and coarse-grained (three classes: Negative, Neutral, Positive) emotion classification on keyframes. To enable cross-modal emotion understanding without fine-tuning or labeled data, we propose a novel multi-frame fusion prompting strategy. Experimental results show that GPT-4o-mini achieves 50% average precision on the seven-class task and 64% on the three-class task. Multi-frame fusion significantly improves robustness and reduces annotation overhead. This work departs from conventional supervised paradigms, demonstrating that LLMs—when guided by carefully designed visual prompting—can effectively infer emotional states from uncurated, naturalistic video content. It establishes a new pathway for automated, low-resource emotion analysis in real-world settings.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Computer Vision: Large Vision ModelsNatural Language Processing: (Large) Language Models

Application Category

Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labelingUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendationSearch and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

This study investigates the feasibility and performance of using large language models (LLMs) to automatically annotate human emotions in everyday scenarios. We conducted experiments on the DailyLife subset of the publicly available FERV39k dataset, employing the GPT-4o-mini model for rapid, zero-shot labeling of key frames extracted from video segments. Under a seven-class emotion taxonomy ("Angry,""Disgust,""Fear,""Happy,""Neutral,""Sad,""Surprise"), the LLM achieved an average precision of approximately 50%. In contrast, when limited to ternary emotion classification (negative/neutral/positive), the average precision increased to approximately 64%. Additionally, we explored a strategy that integrates multiple frames within 1-2 second video clips to enhance labeling performance and reduce costs. The results indicate that this approach can slightly improve annotation accuracy. Overall, our preliminary findings highlight the potential application of zero-shot LLMs in human facial emotion annotation tasks, offering new avenues for reducing labeling costs and broadening the applicability of LLMs in complex multimodal environments.

Problem

• Large language models for emotion annotation
• Zero-shot labeling in daily scenarios
• Multi-frame approach to enhance accuracy

Innovation

• Utilizes GPT-4o-mini for zero-shot labeling
• Employs multi-frame strategy for accuracy
• Explores ternary emotion classification enhancement