UNMATCH: Selective Unbalanced Token-Patch Matching for Forensic Image-Claim Verification

2026-09-24

AI Summary

This study addresses the challenge that existing methods struggle to capture critical local mismatches in the forensic verification of images paired with misleading claims. To this end, we propose a directional multi-scale coverage representation framework. This approach innovatively introduces a compact directional multi-scale local affinity representation to effectively preserve decisive local mismatch information. Furthermore, by integrating a threshold-supported ratio with a lightweight global-local classifier, it achieves precise image-text correspondence detection. Experimental results demonstrate that the proposed model attains a Macro-F1 score of 69.82% and a balanced accuracy of 71.05% on the Fauxtography benchmark, significantly outperforming existing baseline methods.

Abstract

Contextual image misuse pairs an image with a misleading claim. We study image-claim correspondence in fact-checked pairs containing out-of-context reuse, visual manipulation, or both. Existing pair-based detectors often compress the two modalities into a global compatibility score or learn a highly flexible interaction module, which can obscure a decisive local mismatch. We introduce directional multiscale coverage, a compact representation that summarizes local image-claim affinity in both directions and at three spatial scales. At each scale, each direction is summarized by its mean, lower quartile, and two thresholded support ratios; the signed difference between directional means completes a nine-dimensional scale descriptor. Concatenating the three scales yields a compact local representation for a lightweight global-local classifier. Under leakage-aware three-fold, three-seed evaluation on the Snopes subset of the Fauxtography benchmark, UNMATCH achieves 69.82 Macro-F1 and 71.05 balanced accuracy, exceeding the MCOT adaptation by 2.60 and 2.16 points. A matched-reassigned intervention shows that breaking the observed pairing lowers coverage and increases both discrepancy and false-pair probability.

Problem

• forensic image-claim verification
• out-of-context image reuse
• visual manipulation
• multimodal mismatch detection
• fact-checking

Innovation

• Forensic Image-Claim Verification
• Directional Multiscale Coverage
• Token-Patch Matching
• Out-of-context Misuse
• Global-local Classifier