Modality Agreement- and Conflict-Aware Prototype Hypergraph Learning for Multimodal Intent Understanding

2026-08-04

AI Summary

This work addresses a critical limitation in existing multimodal intent understanding approaches, which often overlook conflicting signals across modalities—such as positive linguistic content paired with negative vocal or facial cues—despite their high discriminative value. To tackle this, the authors propose MACH, a novel framework that explicitly models modality agreement and conflict as two distinct, learnable, and reusable relational structures. MACH employs hierarchical prototype hypergraphs to construct sparse prototypes for both consistency and conflict, and introduces a feature-level, sample-adaptive arbitration mechanism to fuse these signals, preserving discriminative discrepancies while suppressing irrelevant noise. Extensive experiments demonstrate that MACH significantly outperforms state-of-the-art methods across multiple benchmarks, and ablation studies confirm the effectiveness of its hierarchical composition, prototype semantic refinement, and arbitration design.

Abstract

Multimodal intent recognition requires understanding not only what textual, acoustic, and visual signals share, but also how they disagree. Such disagreement is frequently class-informative; for example, lexical positivity accompanied by incongruent vocal or facial behavior may indicate sarcasm or taunting, yet most fusion methods either encourage modality alignment or treat inconsistency as uncertainty to be suppressed. We propose MACH (Modality Agreement- and Conflict-aware prototype Hypergraph), a hierarchical prototype-hypergraph framework that represents multimodal agreement and conflict as distinct, recurring relational structures. MACH progressively composes unimodal representations into bimodal and trimodal abstractions. At each applicable level, modality-composition anchors activate sparse agreement prototype hypergraphs that capture reusable consensus patterns, while a separate conflict pathway maps cross-modal discrepancies to dedicated conflict prototype hypergraphs. The two pathways are combined through a feature-wise, sample-adaptive arbitration mechanism, enabling the model to preserve informative disagreement while suppressing incidental modality noise. A progressive optimization strategy stabilizes the interdependent hierarchy before joint agreement-conflict learning. Experiments on benchmark datasets demonstrate the effectiveness of the proposed formulation, while component and robustness analyses validate the distinct roles of hierarchical composition, prototype-mediated semantic refinement, and agreement-conflict arbitration.

Problem

• multimodal intent recognition
• modality agreement
• modality conflict
• sarcasm detection
• cross-modal discrepancy

Innovation

• prototype hypergraph
• modality conflict
• hierarchical fusion
• agreement-aware learning
• multimodal intent recognition