CALM-AH: An ABAW11-Calibrated Multimodal Ensemble with Reliability-Gated Multi-Expert Consensus for Video-Level Ambivalence and Hesitancy Recognition

2026-07-31

AI Summary

This work addresses the task of recognizing ambivalence and hesitation (A/H) behaviors in naturalistic interview videos by proposing a Reliability-Gated Multi-Expert Consensus (RG-MEC) mechanism. The approach integrates textual, acoustic, visual, and behavioral statistical features across 15 multimodal combinations, generating initial predictions through Macro-F1–oriented threshold optimization and hard voting ensemble. To refine these predictions, RG-MEC employs an anchor retention strategy and a three-expert consensus correction policy, modifying outputs only when complementary experts agree. Additionally, GPT-based semantic validation and AffectGPT-driven affective analysis are incorporated to enhance decision reliability. Evaluated on the ABAW11 dataset, the RG-MEC system achieves a Macro-F1 score of 0.7771, significantly outperforming the baseline CALM-AH (0.7525), thereby demonstrating the efficacy of multimodal synergy and gated consensus mechanisms in A/H behavior recognition.

Abstract

Ambivalence and hesitancy (A/H) are subtle behavioural states that may be expressed through language, voice, facial activity, and other non-verbal cues. The ABAW11 A/H Video Recognition Challenge asks systems to assign a binary A/H label to each naturalistic interview video. Performance is measured using Macro-F1 so that recognition of both A/H and No-A/H samples receives equal importance. We present CALM-AH, a multimodal ensemble that combines textual, acoustic, visual, and derived behavioural-statistical features. We construct 15 non-empty combinations of these feature branches. For each combination, we select the best of three classifier families using validation binary cross-entropy and optimise its decision threshold for validation Macro-F1. The resulting binary decisions are combined using fixed hard-voting weights transferred from BROTHER. We further introduce Reliability-Gated Multi-Expert Consensus(RG-MEC), an anchor-preserving decision-level ensemble that combines an initial prediction with three complementary correction experts: CALM-AH, AffectGPT, and a GPT-based semantic verifier. The initial system provides the default prediction. Its label is overridden only when all three correction experts unanimously support the same alternative class; otherwise, the anchor prediction is retained. This unanimity-gated design limits the influence of isolated expert errors while permitting bidirectional correction when task-specific, multimodal-affective, and semantic-pragmatic evidence are fully consistent. On the participant-disjoint ABAW11 dataset, CALM-AH achieves a Macro-F1 of 0.7525, and the complete RG-MEC system achieves 0.7771.

Problem

• ambivalence
• hesitancy
• video-level recognition
• multimodal behavior
• ABAW11

Innovation

• multimodal ensemble
• Reliability-Gated Multi-Expert Consensus
• Macro-F1 optimization
• decision-level fusion
• ambivalence and hesitancy recognition