ViSR-KGC: Visual Subgraph Reasoning with Vision-Language Models for Multimodal Knowledge Graph Completion

2026-08-06

AI Summary

This work addresses the challenges of multimodal knowledge graph completion under relation sparsity, where conventional embedding methods suffer performance degradation and large language models lose critical topological and visual information through linearization. To overcome these limitations, the paper proposes ViSR-KGC, which uniquely visualizes query-relevant subgraphs as images and constructs a unified prompt by integrating entity images, textual descriptions, and candidate answers to guide reasoning with a vision-language model. By synergistically combining multimodal embedding learning, subgraph extraction, graph layout visualization, and prompt engineering, ViSR-KGC effectively leverages global topological structure, local multimodal evidence, and pretrained commonsense knowledge. The method significantly outperforms existing embedding-based and large language model approaches in sparse relational scenarios.

Abstract

Knowledge graph completion (KGC) aims to infer missing entities or relations from incomplete graph structures, and has evolved into multimodal knowledge graph completion (MMKGC), where entities are associated with multiple modalities such as text and images. Traditional representation learning approaches follow the embedding-based paradigm and may struggle when relation-specific evidence is limited. Meanwhile, LLM-based reasoning methods typically linearize graph structures into textual prompts, which obscures structural topology and neglects vital visual information. While vision-language models (VLMs) excel at multimodal reasoning, they cannot natively interpret structured graph topology, particularly when it comes to knowledge graphs where nodes and edges carry complex semantics. To bridge this gap, we propose ViSR-KGC, a visual subgraph reasoning approach for KGC. It integrates three complementary capabilities to capture semantic correlations: identifying global topology dependencies via representation learning, analyzing local multimodal evidence using VLMs, and providing necessary commonsense knowledge inherent in pre-trained models. Based on learned multimodal embeddings, our framework first extracts a compact and query-aware subgraph from the MMKG. Then, this subgraph is transformed into a visually interpretable image using a layout strategy selected through empirical comparison.Finally, the visualized subgraph, entity images, textual descriptions, and candidate answers are combined into a unified prompt, enabling the VLM to infer the missing entity.

Problem

• multimodal knowledge graph completion
• visual subgraph reasoning
• vision-language models
• graph topology
• knowledge graph completion

Innovation

• visual subgraph reasoning
• vision-language models
• multimodal knowledge graph completion
• graph visualization
• structured reasoning