Towards Grounded GI Endoscopy VQA via Multi-Task Learning on Small VLMs

2026-07-29

AI Summary

This work addresses the lack of interpretability and visual grounding in gastrointestinal endoscopy visual question answering (VQA) by proposing a lightweight multi-task fine-tuning strategy. Leveraging low-rank adaptation (LoRA), a small vision-language model is fine-tuned to jointly perform the primary VQA task alongside weakly supervised auxiliary tasks—namely, image region localization and description. The approach exploits existing polyp masks and Grad-CAM to generate weak supervision signals without requiring extensive additional annotations. Evaluated on the Kvasir-VQA-x1 dataset, the method significantly improves accuracy on both in-distribution and out-of-distribution test sets while implicitly enhancing alignment between model predictions and relevant image regions, thereby increasing clinical trustworthiness.

Abstract

Gastrointestinal (GI) endoscopic image analysis has shifted from single-label classification toward visual question answering (VQA), where a model must answer free-form clinical questions about an image. While recent vision-language models (VLMs) achieve promising answer accuracy on this task, clinical adoption also requires the model's internal representations to reflect the visual evidence behind its answers. We propose a simple multi-task fine-tuning recipe that constructs auxiliary grounding and description tasks from an existing VQA dataset with minimal additional annotation: expert-annotated polyp masks are reused directly, while a GI-domain pretrained classifier with Grad-CAM localization provides weak supervision for finding categories that lack ground-truth masks. Three small VLM backbones are fine-tuned with low-rank adaptation under matched VQA-only and multi-task recipes on Kvasir-VQA-x1, and we show consistent accuracy gains together with improved implicit alignment between answer tokens and the relevant image region, evaluated on both in-distribution and out-of-distribution data.

Problem

• GI endoscopy
• visual question answering
• grounding
• vision-language models
• clinical interpretability

Innovation

• multi-task learning
• visual grounding
• vision-language models
• low-rank adaptation
• endoscopic VQA