DiffImaginE: Imagine to Verify Entity Types with Diffusio

2026-08-03

AI Summary

Existing multimodal named entity recognition approaches struggle to effectively model the joint support of textual and visual evidence for entity typing and lack explicit probabilistic semantics. This work introduces diffusion models into multimodal entity type verification for the first time, proposing a conditional latent diffusion inference framework. The framework employs a type-conditional denoiser to predict noise injected into normalized latent representations and constructs a negative log-likelihood surrogate consistent with the evidence lower bound (ELBO) from denoising errors to rank competing entity types. The method integrates classifier-free guidance, Min-SNR weighting during training, and dual sampling to enhance performance and stability. Evaluated on Twitter-2015 and Twitter-2017 datasets under identical encoders and evaluation protocols, it significantly outperforms matched deterministic baselines, achieving consistent gains. Ablation studies and significance tests confirm the effectiveness of each component.

Abstract

Multimodal named entity recognition (MNER) determines whether each candidate span and entity-type hypothesis is supported by joint textual and visual evidence. Existing imagine-and-compare verifiers map each (span, type) pair to one predicted visual feature, compressing diverse visual realisations into a single prototype and providing a compatibility score without explicit probabilistic semantics. We introduce DiffImaginE, which formulates MNER type verification as conditional latent diffusion inference. Given span-localised visual evidence, a type-conditioned denoiser predicts noise injected into its standardised latent. The resulting denoising error provides an ELBO-consistent surrogate for type-conditional negative log-likelihood, allowing competing type hypotheses to be ranked by how well they explain the observation. DiffImaginE retains a standard multimodal encoder stack and replaces the deterministic verifier with a classifier-free-guided diffusion scorer trained using Min-SNR weighting. We directly supervise per-type diffusion scores as classification logits, learn aggregation across noise levels, and use antithetic sampling to reduce Monte Carlo comparison variance. Our analysis shows that classifier-free guidance sharpens the induced type posterior and characterises when antithetic pairing reduces variance at equal denoiser cost. Experiments on Twitter-2015 and Twitter-2017 show consistent gains over a matched deterministic ImaginE control under the same encoder, auxiliary objectives, and evaluation protocol, supported by ablations and paired significance tests.

Problem

• multimodal named entity recognition
• entity type verification
• visual evidence
• probabilistic semantics
• conditional latent diffusion

Innovation

• conditional latent diffusion
• multimodal named entity recognition
• classifier-free guidance
• denoising error
• antithetic sampling