Adaptive Emotional Video Captioning via Affective Heterogeneous Graph Reasoning and Multi-task Joint Learning

2026-07-31

AI Summary

Existing methods for emotional video captioning are limited by global emotion vectors or rigid hierarchical priors, struggling to model ambiguous, mixed, and dynamically evolving emotions, while coarse-grained emotion errors often irreversibly suppress fine-grained affective words. To address these issues, this work proposes a soft emotional heterogeneous graph that jointly models category-level emotion nodes and word-level sentiment nodes. By incorporating continuous emotion bias through soft-gated graph attention and integrating a causal language decoder with dual-head emotion distribution supervision, the approach enables multi-task joint optimization of caption generation and emotion prediction. Replacing hard-masked tree-structured priors with a recoverable soft graph structure, the method supports coexisting emotions and error-resilient expression, significantly enhancing the nuance, diversity, and robustness of emotional descriptions while preserving visual factual accuracy.

Abstract

Emotional video captioning (EVC) aims to describe a video with both factual correctness and affective expressiveness. It requires a model to perceive subtle, ambiguous, and temporally varying emotional cues and translate them into natural language without weakening objective visual content. Existing methods have progressively introduced contextual attention, emotion interpretation, emotion priors, dynamic emotion perception and emotion-cause reasoning. Nevertheless, most of them still depend on either global emotion vectors or rigid hierarchical priors. In recent methods, the tree-structured emotion prior establishes a coarse-to-fine connection between psychological emotion categories and daily emotion words, but its hard subordinate masking may irreversibly suppress correct lexical emotions once the coarse category prediction is inaccurate. It is also limited in representing mixed or overlapping emotions that frequently occur in real videos. To address the issues, we propose SAGML, an adaptive EVC framework via affective heterogeneous graph and multi-task language modeling. Instead of treating the emotion prior as a discrete tree, SAGML constructs a soft affective heterogeneous graph containing catalog-level emotion nodes and lexical-level emotion word nodes. The soft gate is injected into video-to-emotion graph attention as a continuous bias, allowing visually supported lexical emotions to remain recoverable rather than being removed by a hard mask. The resulting affective representation is fed together with visual tokens into a causal language decoder, while dual catalog and lexical heads impose explicit emotion distribution learning on the prompt hidden states. The overall model is trained with a joint objective that combines autoregressive caption generation and emotion distribution supervision. SAGML provides an error-resilient and multi-emotion-aware baseline for EVC.

Problem

• Emotional Video Captioning
• Affective Heterogeneous Graph
• Mixed Emotions
• Emotion Prior
• Lexical Emotion

Innovation

• affective heterogeneous graph
• soft gating mechanism
• multi-task joint learning
• emotional video captioning
• emotion distribution supervision