Objective-Aligned Direct Answer SFT for Robust Multi-Frame Medical VQA

2026-07-29

AI Summary

This work addresses the limitations of existing adaptation methods in multi-frame medical visual question answering, which are often overly complex and lack consistent optimization toward the answer objective. The authors propose a Direct Answer Supervised Fine-Tuning (SFT) paradigm, employing a decoder-only architecture, fixed data splits, matched computational budgets, multiple random seeds, and post-hoc calibration to enable a rigorously controlled evaluation of adaptation strategies. Experiments demonstrate that this approach substantially improves report accuracy with high stability on MedGemma-1.5-4B and generalizes effectively to Qwen2.5-VL-3B. Furthermore, calibration significantly enhances confidence estimation. These findings challenge the prevailing reliance on intricate mechanisms, underscoring that simple fine-tuning aligned directly with the target objective yields more robust performance.

Abstract

Multi-frame medical VQA appears to reward increasingly complex adaptation: controller-style inference, localization-aware reranking, static hard-negative mixing, and staged continuation all appear plausible from first principles. We test a simpler competing hypothesis on MedFrameQA: methods that remain tightly aligned with the benchmark's final answer objective should be the strongest \emph{robust} adaptation family once evaluation is controlled across fixed splits, matched budgets, repeated seeds, and calibration. We compare controller-based methods, scaffold evolution, static mixed supervision, continuation-heavy variants, and direct answer-only supervised fine-tuning (SFT). The strongest robust family is direct decoder-only answer SFT on MedGemma-1.5-4B. Empirically, this family yields substantial improvements in held-out report accuracy over frozen baselines while remaining remarkably stable across repeated seeds and matched controls, ensuring our claims reflect true family-level robustness rather than an isolated hyperparameter peak. Furthermore, post-hoc calibration effectively repairs confidence estimation without compromising accuracy, and the core approach transfers consistently to secondary backbones like Qwen2.5-VL-3B. The main result is therefore not that a complex auxiliary mechanism wins, but that objective-aligned direct answer SFT is the strongest robust adaptation family we found for MedFrameQA. By establishing this strong, minimalist baseline, we hope to redirect community focus toward fundamentally robust optimization rather than architectural complexity.

Problem

• Medical VQA
• multi-frame
• robust adaptation
• objective alignment
• supervised fine-tuning

Innovation

• Direct Answer SFT
• Objective Alignment
• Robust Medical VQA
• Multi-frame VQA
• Minimalist Adaptation