CXR-Retrieve: Compositional Text-to-Image Retrieval in Chest Radiography

2026-07-30

AI Summary

Existing text–image retrieval methods for chest X-rays struggle with clinical queries involving compositional logic (e.g., “atelectasis and no pneumonia”) because they optimize only holistic report-level matching and ignore logical constraints among pathological assertions. To address this, this work introduces CXR-Retrieve—the first structured retrieval benchmark tailored for compositional logic, including negation—and proposes a label-aware contrastive fine-tuning objective that explicitly models pathological co-occurrence and negation relationships to align assertion consistency and distinguish contradictions. Built upon CXR-CLIP using MIMIC-CXR-JPG, the benchmark comprises 5,159 test images and 145 compositional queries. The proposed approach substantially outperforms existing baselines, achieving an 8.5% gain in Precision@5 on dual-pathology queries and a 22.0% improvement on negation queries.

Abstract

Large chest radiography archives are difficult to search because most studies are paired only with free-text reports rather than structured clinical annotations. Vision-language models offer a natural interface for text-to-image retrieval, but current biomedical models are primarily optimized for report-to-image matching rather than for satisfying short clinical search queries. This creates an objective mismatch: a model may retrieve images related to words in the query while failing to satisfy the full clinical constraint, especially for conjunctions and negations such as ``atelectasis and no pneumonia.'' We introduce CXR-Retrieve, a structured benchmark for compositional chest X-ray text-to-image retrieval. The benchmark contains 5,159 test images from the official test-split of MIMIC-CXR-JPG and 145 textual queries spanning single and conjunction findings, both positive and negative. Relevance is defined by whether a retrieved image satisfies all asserted pathology constraints, rather than by whether it matches a paired report. We further propose a label-aware contrastive fine-tuning objective for clinical retrieval. Our method attracts image-text pairs with compatible asserted pathology constraints, including shared confirmed absences, while explicitly repelling contradictory pairs. Starting from the in-domain CXR-CLIP checkpoint, our method improves Precision@5 over CXR-CLIP by 8.5 percentage points on two-pathology conjunctions and by 22.0 percentage points on negation queries. These results show that reliable chest X-ray retrieval requires training objectives that model not only which findings are mentioned, but also how they are clinically asserted.

Problem

• compositional retrieval
• chest radiography
• text-to-image retrieval
• clinical negation
• pathology constraints

Innovation

• compositional retrieval
• label-aware contrastive learning
• chest radiography
• negation handling
• clinical text-to-image retrieval