UniHEAR: Unified Heterogeneous-Source Attentive Retrieval for Knowledge-Based Visual Question Answering

2026-08-02

AI Summary

This work addresses the limitations in knowledge-based visual question answering, specifically the single-source retrieval bottleneck and the neglect of retrieval provenance during reranking. To overcome these issues, the authors propose UniHEAR, a unified framework for multi-source entity retrieval and reranking. The method constructs coarse-grained retrieval descriptors, introduces a retrieval-guided modality attention gating mechanism, and employs entropy-weighted fusion of multi-source priors to enable lightweight, heterogeneous source collaboration. The training strategy integrates a dual-tower architecture with contrastive learning and a modality-preserving auxiliary loss. Evaluated on E-VQA and InfoSeek benchmarks, UniHEAR achieves state-of-the-art performance, improving Recall@1 by 6.7% and 1.2%, respectively, significantly outperforming existing approaches.

Abstract

Knowledge-Based Visual Question Answering (KB-VQA) requires retrieving relevant entity knowledge from external sources to answer visually grounded questions. Existing retrieval-augmented systems suffer from two critical limitations. First, relying on a single retrieval modality creates a Single-Source Retrieval Bottleneck, missing ground-truth entities that are only accessible through complementary sources. Second, dual-tower pointwise rerankers suffer from Retrieval-Source-Blind Reranking, as they overlook retrieval origins and candidate-level retrieval priors, leading to redundant modality reliance. To address these challenges, we propose UniHEAR, a unified lightweight framework for heterogeneous-source entity retrieval and reranking. UniHEAR constructs a Coarse Retrieval Descriptor for each candidate entity, and introduces Retrieval-Guided Attentive Modality Gating to condition modality attention weights on this descriptor, further complemented by Entropy-Weighted Source Fusion of coarse retrieval priors. A hybrid training strategy combining contrastive learning with an auxiliary modality-preserving loss unifies entity-level and section-level retrieval within a single model. Extensive experiments on E-VQA and InfoSeek demonstrate that UniHEAR achieves state-of-the-art retrieval and VQA performance, improving Recall@1 by 6.7 and 1.2 points over the strongest baselines while maintaining a lightweight reranking architecture. Code and model are available at https://github.com/iven-luo/UniHEAR.

Problem

• KB-VQA
• heterogeneous-source retrieval
• single-source bottleneck
• retrieval-source-blind reranking
• entity knowledge retrieval

Innovation

• heterogeneous-source retrieval
• retrieval-guided attention
• modality gating
• entropy-weighted fusion
• unified reranking