Bayesian Data Reweighting Improves Multimodal Retrieval for Knowledge-Based Visual Question Answering

2026-08-03

AI Summary

This work addresses a critical limitation in existing multimodal contrastive learning approaches, which treat all non-matching query–document pairs uniformly as negative samples, thereby ignoring potential semantic relevance and introducing false negatives that degrade retrieval performance. To overcome this, the paper proposes the first Bayesian data reweighting framework tailored for multimodal retrieval, modeling sample importance as a latent variable. By leveraging closed-form posterior inference under conjugate priors and a stochastic EM algorithm, the method dynamically identifies and downweights false negative samples during training. This approach transcends the conventional “one-size-fits-all” treatment of negative samples in contrastive learning and consistently achieves significant gains in retrieval accuracy across three distinct retrievers and seven knowledge-intensive visual question answering benchmarks, demonstrating both its effectiveness and strong generalization capability.

Abstract

Multimodal retrievers are essential for knowledge-based visual question answering, where they retrieve external evidence for image-question pairs. However, existing contrastive training methods typically treat all unmatched query-document pairs as equally informative negatives, which is problematic because many unmatched documents may still be semantically relevant or partially useful. We propose Bayesian Data Reweighting, a probabilistic framework that models query-document importance as latent variables and adaptively infers posterior weights to downweight likely false negatives. With closed-form posterior updates under conjugate priors and stochastic EM optimization, our method consistently improves retrieval accuracy across three retrievers and seven knowledge-based VQA benchmarks.

Problem

• multimodal retrieval
• knowledge-based VQA
• false negatives
• contrastive training
• visual question answering

Innovation

• Bayesian Data Reweighting
• multimodal retrieval
• knowledge-based VQA
• false negative downweighting
• stochastic EM