🤖 AI Summary
Existing methods for detecting out-of-context (OOC) misinformation—where authentic images are falsely paired with misleading textual claims—largely rely on external evidence but neglect the evidential stance (supportive or refuting) toward the claim. Method: We propose a stance-aware multimodal detection framework featuring: (i) a novel Stance Extraction Network (SEN) that models evidential stance as a critical deviation signal; (ii) fine-grained semantic alignment via named-entity co-occurrence–based support-refute scoring; and (iii) an end-to-end trainable architecture integrating joint text-image encoding. Contribution/Results: Evaluated on large-scale public benchmarks, our approach surpasses state-of-the-art methods, achieving up to a 3.2% absolute accuracy gain. It significantly improves both robustness and interpretability in OOC misinformation detection by explicitly modeling and leveraging stance information.
📝 Abstract
Mis- and disinformation online have become a major societal problem as major sources of online harms of different kinds. One common form of mis- and disinformation is out-of-context (OOC) information, where different pieces of information are falsely associated, e.g., a real image combined with a false textual caption or a misleading textual description. Although some past studies have attempted to defend against OOC mis- and disinformation through external evidence, they tend to disregard the role of different pieces of evidence with different stances. Motivated by the intuition that the stance of evidence represents a bias towards different detection results, we propose a stance extraction network (SEN) that can extract the stances of different pieces of multi-modal evidence in a unified framework. Moreover, we introduce a support-refutation score calculated based on the co-occurrence relations of named entities into the textual SEN. Extensive experiments on a public large-scale dataset demonstrated that our proposed method outperformed the state-of-the-art baselines, with the best model achieving a performance gain of 3.2% in accuracy.
