Attention Consistent Longitudinal Medical Visual Question Answering Guided by Vision Foundation Models

2026-06-03

AI Summary

This work addresses the challenge of reasoning about anatomical changes across temporal phases in longitudinal medical visual question answering by proposing an attention-guided encoder–decoder architecture. The method employs lightweight affine registration to pre-align current and reference images and integrates a frozen DINO-v2 mask generator with a trainable adaptive masking module to extract salient regions. A multimodal Transformer decoder then fuses image and text features to generate answers. Innovatively, multiple auxiliary losses—including mask reconstruction, Gram consistency, and KoLeo uniformity—are jointly optimized to enhance the geometric structure of learned representations while sharpening change-related signals. Evaluated on the Medical-Diff-VQA benchmark, the model significantly outperforms existing approaches and achieves intrinsic interpretability through shared saliency masks.

Abstract

Longitudinal medical visual question answering (VQA) requires reasoning about anatomical differences between an image of a current time point and an image of a referred time point. We propose an attention-guided encoder-decoder for this task with chest X-rays. Instead of conventional direct contrast, we propose to include a lightweight affine registration module to reduce nuisance motion by co-registering the current image to the reference image with a small registration regularizer. The registered image pair is fed into the image encoder, followed by a frozen DINO-based mask generator and a trainable adaptive mask generator to produce masks applied to the original image pairs. The masked image pairs are again fed into the image encoder and concatenated with text features as the input to a multimodal transformer-based decoder to generate final answers. To facilitate learning stabilization and clarify the change signal, inspired by DINO-v3, we include additional auxiliary objectives, including a mask rebuilding loss, a pairwise Gram-style consistency loss, and a KoLeo uniformity loss, which enhances the geometry of the representation. On the Medical-Diff-VQA benchmark, the model delivers strong BLEU, ROUGE-L, CIDEr, and METEOR scores while offering intrinsic interpretability through the shared saliency mask. These results support saliency-conditioned generation with mild pre-alignment as a principled framework for longitudinal reasoning in medical VQA. Our training strategy also illustrates the potential of a paradigm in utilizing image foundation models in biomedicine: optimizing both supervised and unsupervised learning objectives simultaneously.

Problem

• longitudinal medical VQA
• anatomical differences
• medical image reasoning
• temporal comparison
• visual question answering

Innovation

• longitudinal medical VQA
• affine registration
• saliency mask
• vision foundation models
• auxiliary learning objectives