VietMEAgent: Culturally-Aware Few-Shot Multimodal Explanation for Vietnamese Visual Question Answering

2025-11-12

AI Summary

Existing visual question answering (VQA) systems exhibit limited cultural understanding of Vietnamese content due to insufficient representation of cultural knowledge and opaque reasoning processes. Method: We propose the first explainable VQA framework tailored to Vietnamese culture, integrating culture-aware multimodal reasoning with a dual-modal explanation mechanism: (1) a structured program generation module jointly predicting answers and performing cultural reasoning; (2) an attention-driven visual evidence extraction module and a knowledge-base-augmented textual rationale generation module; and (3) a Vietnamese culture-specific knowledge base supporting few-shot cultural comprehension. Contribution/Results: Evaluated on a newly constructed Vietnamese Cultural VQA dataset, our framework significantly improves cultural accuracy of answers and readability of explanations, thereby enhancing user trust in AI decisions and deepening cross-cultural cognitive understanding.

Technology Category

• Computer Vision: Visual Reasoning & Symbolic Representations
• Natural Language Processing: Question Answering
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Web Mining and Content Analysis: Community question answering

Abstract

Contemporary Visual Question Answering (VQA) systems remain constrained when confronted with culturally specific content, largely because cultural knowledge is under-represented in training corpora and the reasoning process is not rendered interpretable to end users. This paper introduces VietMEAgent, a multimodal explainable framework engineered for Vietnamese cultural understanding. The method integrates a cultural object detection backbone with a structured program generation layer, yielding a pipeline in which answer prediction and explanation are tightly coupled. A curated knowledge base of Vietnamese cultural entities serves as an explicit source of background information, while a dual-modality explanation module combines attention-based visual evidence with structured, human-readable textual rationales. We further construct a Vietnamese Cultural VQA dataset sourced from public repositories and use it to demonstrate the practicality of programming-based methodologies for cultural AI. The resulting system provides transparent explanations that disclose both the computational rationale and the underlying cultural context, supporting education and cultural preservation with an emphasis on interpretability and cultural sensitivity.

Problem

• Addresses cultural limitations in Vietnamese Visual Question Answering systems
• Develops multimodal explainable framework for cultural object detection
• Enhances interpretability through visual evidence and cultural knowledge integration

Innovation

• Cultural object detection backbone for Vietnamese VQA
• Structured program generation for coupled prediction-explanation
• Dual-modality explanation combining visual and textual rationales