Learning to Detect Cross-Modal Negation: An Analysis of Latent Representations and an Attention-Based Solution

2026-07-20

AI Summary

This study addresses the challenge that existing vision-language models struggle to detect high-level semantic concepts such as negation, primarily due to the non-separability of negation classes in their latent spaces. The work reveals, for the first time, that textual negation can be represented independently, whereas visual negation is inherently dependent on linguistic context. To tackle this issue, the authors propose a novel cross-attention architecture that explicitly models inter-modal dependencies. The approach integrates self-supervised video representations from JEPA2 with automatic annotations generated by Qwen2.5-VL, and is evaluated on a dataset of 3,222 political video–text pairs. Experimental results demonstrate that the method achieves up to a 7.03% improvement in F1 score over unimodal baselines, advancing multimodal representation learning for temporal negation modeling and semantic alignment.

Abstract

Detecting high-level semantic concepts like negation across modalities remains a challenge for current multimodal systems. We analyze this as a fundamental representation learning problem, providing the first evidence that negation does not form a linearly or non-linearly separable class in the latent spaces of standard vision-language models (VLMs). We demonstrate that pretrained embeddings primarily encode modality-specific features, lacking a generalizable negation signal. To overcome this, we propose a novel cross-modal attention architecture that explicitly models inter-modal dependencies, achieving performance gains of up to +7.03% F1 over unimodal baselines. Our analysis reveals a key asymmetry: while textual negation often appears independently, visual negation is semantically dependent on linguistic context, a finding validated through our statistical analysis of 3,222 political video-text pairs automatically annotated via \textsc{Qwen2.5-VL}. By combining this analysis with self-supervised video representations (JEPA2), we advance the modeling of temporal negation. This work provides new methods and insights for learning robust, semantically-aligned representations in multimodal systems.

Problem

• cross-modal negation
• multimodal representation
• vision-language models
• semantic alignment
• latent representation

Innovation

• cross-modal negation
• latent representation analysis
• attention-based architecture
• vision-language models
• self-supervised video representation