Cognition-Oriented Emotion Tracing from Causes to Consequences in Real-World Social Scenes

2026-10-08

AI Summary

This study addresses the limitations of affective computing in overlooking cognitive processes and its difficulty in tracing emotional causal chains within authentic social scenarios by proposing the TRACE framework. This work is the first to formalize emotion as a three-stage cognitive model comprising "condition-influence-effect," constructs a multimodal emotion tracking benchmark, and designs TRACER, a structured reasoning method grounded in explicit premises. By integrating multimodal large language models with structured graph reasoning, TRACER comprehensively surpasses existing baselines across five tasks, significantly narrows the human-machine performance gap, and effectively mitigates hallucination issues in large models. Ultimately, this research establishes a novel paradigm for understanding dynamic emotional evolution.

Abstract

Affective computing has progressed from categorical emotion recognition to open-ended affective analysis with large multimodal models. Yet affective science describes emotion as an unfolding process shaped by appraisal, regulation, and social interpretation, which remains underexplored computationally. We propose TRACE, a cognition-oriented framework that formalizes an affective episode through three interrelated stages: Condition, Affect, and Effect, integrating observable cues with cognitive factors such as internal stance and regulation of emotional display. Based on this formulation, TRACE-Bench evaluates multimodal models in real-world social scenes through five tasks spanning grounded affect recognition, regulation decoding, cause reasoning, effect reasoning, and full-chain reconstruction, with 3,746 structured question-answer pairs over 646 videos. A matched human-model comparison reveals a substantial performance gap, while affect-specialized models also generally lag behind general-purpose MLLMs. Model outputs show recurring failures, including treating displayed behavior as genuine feeling and fabricating unsupported events during long-chain generation. We further propose TRACER, a cognition-grounded structured reasoning method that couples each inference with explicit premises from factual observations, cognitive appraisals, and established upstream conclusions, forming a traceable graph of intermediate and target conclusions. TRACER outperforms all evaluated model baselines on each of the five tasks. Project page: https://cogaffc.github.io/TRACE

Problem

• Affective Computing
• Emotion Tracing
• Cognitive Appraisal
• Multimodal Large Language Models
• Social Scene Understanding

Innovation

• Cognition-Oriented Framework
• Affective Computing
• Multimodal Benchmark
• Structured Reasoning
• Traceable Inference Graph