Analyzing and Improving Fine-grained Preference Optimization in Medical LVLMs

2026-06-10

AI Summary

This work addresses critical limitations in medical large vision-language models—namely, deficiencies in factual consistency, visual grounding, and clinical alignment—exacerbated by existing preference optimization methods that rely on coarse-grained rewards, suffer from distributional shift, and lack visual constraints. To overcome these challenges, the authors propose a fine-grained online alignment framework that constructs self-feedback preference pairs via minimal-edit generation, selectively correcting only clinically erroneous segments while preserving the original linguistic style. The approach uniquely integrates token-level preference signals with dynamic visual constraints through bidirectional per-token KL regularization and a lesion-perturbation-based visual contrastive grounding mechanism. Evaluated across multiple benchmarks for medical image understanding and clinical text generation, the method demonstrates significant improvements in factual accuracy, visual grounding fidelity, and clinical alignment.

Abstract

Large Vision-Language Models (LVLMs) have achieved strong performance across medical imaging tasks, yet they remain prone to factual inconsistencies, poor visual grounding, and misalignment with clinically meaningful feedback. Existing post-training alignment approaches, including Direct Preference Optimization (DPO) and its variants, face three critical limitations in the medical domain: (1) sequence-level reward signals treat clinically critical tokens identically to generic filler text; (2) reliance on static supervised fine-tuning references as preferred responses introduces an off-policy distribution shift, steering optimization toward stylistic artifacts over clinical correctness; and (3) alignment objectives lack explicit visual grounding constraints, leaving models insensitive to subtle yet diagnostically decisive pathological features. Our method leverages a bidirectional token-wise KL regularizer alongside a visual-contrastive grounding objective that pairs clean and lesion-corrupted images to penalize responses generated without adequate visual evidence. Together, these components form a fine-grained, on-policy alignment framework that constructs preference pairs by minimally editing model-generated outputs, correcting only clinically erroneous spans while preserving the original linguistic style. Extensive experiments across medical imaging tasks and clinical text generation benchmarks validate the effectiveness of our approach.

Problem

• Medical LVLMs
• Preference Optimization
• Visual Grounding
• Clinical Alignment
• Factual Consistency

Innovation

• fine-grained preference optimization
• visual-contrastive grounding
• token-wise KL regularization
• on-policy alignment
• medical LVLMs