Does Vision-Language Pretraining Granularity Matter? A Controlled Evaluation of Vision-Language Objectives Across Chest X-Ray Interpretation Tasks

2026-09-25

AI Summary

This study addresses how the granularity of vision-language pre-training objectives influences chest X-ray interpretation performance, noting that existing research lacks a systematic analysis of the matching mechanisms between granularity and downstream tasks. To bridge this gap, this work integrates global and local contrastive learning with image captioning techniques, systematically evaluating nine pre-training objectives across five downstream tasks via controlled experiments. The findings reveal that no single optimal pre-training objective exists, clarifying the complex interactions among objective types, granularities, and specific tasks. Notably, local objectives significantly benefit abnormality detection, whereas mixed supervision strategies effectively enhance both classification and zero-shot radiology report generation performance.

Abstract

Vision-language pretraining objectives differ in the spatial granularity of their supervision, yet the implications of this distribution for chest X-ray interpretation remain underexplored. We present a controlled study that isolates the pretraining objective: holding the encoder and pretraining data fixed, we train nine objectives spanning global and local contrastive learning, captioning, and their combinations, and evaluate across five chest X-ray tasks of increasing spatial granularity. We find that (i) pretraining granularity aligns with task granularity at the extremes, with local objectives leading on abnormality detection and global objectives on classification; (ii) local objectives are surprisingly competitive on global-level generation and question answering tasks; (iii) the merits of captioning and contrastive learning reverse across granularity levels; and (iv) among combinations, mixing captioning and contrastive supervision is strongest on classification and in distribution generation, while pairing two captioning objectives generalizes best on zero-shot report generation. These results show that no single objective is universally optimal, and that the interaction of objective type, granularity, and task governs downstream performance.

Problem

• Vision-Language Pretraining
• Chest X-Ray Interpretation
• Spatial Granularity
• Pretraining Objectives

Innovation

• Vision-Language Pretraining
• Spatial Granularity
• Chest X-Ray Interpretation
• Contrastive Learning
• Controlled Evaluation