Beyond End-to-End Black Box Mapping: An Intentional Agent Framework for Cognitive-driven Facial Reaction Generation

2026-09-28

AI Summary

This study addresses the reliance of existing facial reaction generation methods on black-box mappings that lack explicit internal states by proposing an Intentional Agent framework for cognition-driven facial reaction synthesis. The approach shifts the paradigm from direct stimulus-response modeling to intermediate-state-based generation, introducing an iterative Inner Thought Flow (ITF) alongside an endogenous dynamic updating mechanism. By integrating an internal dynamics model with an affective mapping module, it constructs a fully interpretable generation pipeline. Experiments on the REACT 2025 dataset demonstrate that the proposed framework achieves an FRDist of 72.39 and significantly outperforms baseline methods in human evaluations. Furthermore, the introduced Reaction Quality Score (RQS) metric exhibits strong alignment with human perception, validating the framework’s advantages in both realism and interpretability.

Abstract

Automatic human-like facial reaction generation (FRG) is essential for building intelligent systems that can engage in human-computer interaction (HCI). While diverse and context-appropriate facial reactions can reflect latent appraisal and affective processes in human interaction, most existing FRG methods rely on end-to-end architectures that directly map speaker behaviours to listener expressions without an explicit intermediate internal state. We reformulate FRG as generation mediated by a structured internal-state process and propose the \textbf{Intentional Agent}, which shifts FRG from direct stimulus-response mapping to stimulus-grounded generation through explicit intermediate states. To represent temporal internal-state evolution, we propose an internal dynamics model that integrates emotional drives with an iterative Inner Thought Flow (ITF) within a structured intermediate state used for subsequent generation. This state can continue to update during conversational silences. Furthermore, to bridge abstract internal states with physiological actions, we formulate FRG as a downstream affective mapping from this latent thought flow to facial expressions. Experiments on the REACT 2025 dataset show an FRDist of 72.39 and an FRDiv of 0.5057; perceptual plausibility is evaluated separately through blinded human ratings. A blinded human evaluation of 96 reactions found no significant difference in mean score between Full and ground truth ($5.527$ vs.\ $5.195$, $p_{\mathrm{Holm}}=.076$), while Full significantly outperformed Event-Triggered and Heuristic-Only (both $p_{\mathrm{Holm}}<.001$). The Reaction Quality Scorer (RQS) correlated strongly with human judgements (Pearson $r=.855$; Spearman $ρ=.821$, both $p<.05$), supporting its use as an automatic metric. These results underscore the immense potential of endogenous dynamics in building highly autonomous, human-like agents.

Problem

• Facial Reaction Generation
• End-to-End Black Box Mapping
• Human-Computer Interaction
• Internal State
• Cognitive-driven

Innovation

• Intentional Agent
• Facial Reaction Generation
• Inner Thought Flow
• Internal Dynamics Model
• Affective Mapping