CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement

2026-08-04

AI Summary

Current vision-language models struggle to simultaneously support the multidimensional clinical demands of chest X-ray analysis, including adjustable-threshold classification, spatial localization, and anatomical measurement. This work proposes the first unified VLM framework that integrates auxiliary discriminative supervision, reward alignment optimization, and native tool calling. Built upon Qwen3-VL-4B-Instruct, the model incorporates a focal loss–based classification head, a composite loss–driven localization head, and a DAPO strategy to achieve mutual enhancement between structured prediction and text generation. It attains state-of-the-art performance across four radiology report generation benchmarks, achieves 94.0% accuracy on ReXVQA, demonstrates spatial localization performance approaching that of specialized detectors, and yields a 43.6-percentage-point average F1 improvement across five measurement-related diseases.

Abstract

A clinically useful chest X-ray system must go beyond fluent report generation: it should classify findings with tunable decision thresholds, localize them spatially, and derive the anatomical measurements upon which many diagnoses depend. Today's Vision-Language Models (VLMs) treat these as separate problems, if they address them at all, leaving a gap between what radiologists need and what generative models provide. We introduce CARE-X, a chest X-ray VLM that narrows this gap by unifying auxiliary discriminative supervision with reward-aligned generation. CARE-X augments its generative backbone with focal-loss classification and composite-loss grounding heads, co-trained alongside the language-modeling objective. This auxiliary supervision produces discriminative diagnostic predictions with tunable decision thresholds and precise spatial localization while also improving report quality, providing evidence that structured prediction and generation reinforce one another. Building on this foundation, Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) leverages task-specific reward signals for report generation, visual question answering (VQA), and spatial grounding, directly optimizing the clinical quality metrics that matter in practice. The result is state-of-the-art performance on the majority of metrics across four report-generation benchmarks, 94.0% VQA accuracy on ReXVQA (+6.0 pp over the next-best baseline), and generative spatial decoding that reaches near parity with dedicated detection heads. Separately, to address measurement-dependent diagnoses, we couple Qwen3-VL-4B-Instruct with native tool-calling capabilities for invoking deterministic measurement tools, while retaining full visual access to the image. This hybrid inference yields +43.6 pp average F1 over perception-only baselines across five measurement-dependent conditions.

Problem

• chest X-ray
• Vision-Language Models
• clinical diagnosis
• anatomical measurement
• spatial localization

Innovation

• auxiliary supervision
• reward-aligned learning
• tool-augmented measurement
• spatial grounding
• vision-language model