Noise-Aware Visual Representation Learning for Medical Visual Question Answering

2026-06-03

AI Summary

This work addresses the vulnerability of visual representations in medical visual question answering (VQA) to noise and minor irrelevant perturbations, which compromises model robustness and performance. To mitigate this issue, the study introduces a denoising autoencoder into the medical VQA pipeline for the first time, reconstructing clean visual embeddings prior to their input into a large language model. The reconstructed features are further processed through an MLP to generate visual prefix tokens, enabling efficient fine-tuning via LoRA. Evaluated on the SLAKE and PathVQA benchmarks, the proposed approach significantly enhances robustness against noisy inputs while maintaining competitive or even superior performance on clean data across multiple metrics, thereby demonstrating its effectiveness and practicality.

Abstract

Medical visual question answering (Med-VQA) has strong potential for clinical decision support by enabling AI models to interpret medical images and answer clinically relevant queries. Recent approaches typically connect off-the-shelf vision encoders with large language models (LLMs) through lightweight mapping networks to reduce computational cost. However, these methods often overlook the importance of handling noise and small irrelevant changes in visual representations. To address these challenges, we propose a noise-aware Med-VQA framework that incorporates a denoising autoencoder before visual embeddings are mapped into the input space of an LLM. The denoising autoencoder is pretrained to reconstruct clean visual embeddings from corrupted inputs, encouraging the model to learn robust visual representations that are less sensitive to noise. The resulting embeddings are then projected into the language model embedding space using a multi-layer perceptron (MLP), forming visual prefix tokens that provide image information to the LLM. To enable efficient adaptation without full retraining, we employ parameter-efficient fine-tuning using low-rank adaptation (LoRA). The proposed method is evaluated on the SLAKE and PathVQA benchmarks. Experimental results show improved robustness to noisy input embeddings while maintaining competitive clean performance across multiple evaluation criteria. These findings suggest that learning more robust visual representations can enhance Med-VQA performance and robustness.

Problem

• Medical Visual Question Answering
• Noise Robustness
• Visual Representation Learning
• Denoising
• Robustness

Innovation

• noise-aware representation
• denoising autoencoder
• medical VQA
• parameter-efficient fine-tuning
• LoRA