RadPRISM: Schema-stratified radiology-report supervision for concept-disentangled image representations and visual grounding

2026-07-31

AI Summary

This work addresses the limitations of existing medical vision–language pretraining models, which suffer from ambiguous clinical concept representation and poor interpretability due to learning in a single shared embedding space. The authors propose the first approach that explicitly leverages the hierarchical structure of clinical radiology concepts as an inductive axis for representation learning. By employing a local large language model to extract concept-level text fragments from radiology reports, they construct dedicated visual subspaces for each clinical concept, enabling natively disentangled, interpretable, and spatially precise alignment. This framework supports fine-grained visual grounding and retrieval, achieving a zero-shot classification macro AUROC of 0.868 (+0.151) on an internal dataset, 4.3× higher visual grounding performance than CARZero in external evaluation, and a top-3 concept retrieval accuracy of 0.78 as assessed by radiologists.

Abstract

Vision-language pretraining learns rich medical image representations from radiology reports, but previous model variants commonly operate within a single shared embedding space, so concept-level structure and interpretability must be recovered post hoc, limiting model transparency and, hence, clinical utility. We introduce RadPRISM, which makes a clinician-defined radiology schema a designated stratification axis: an on-premise large language model extracts per-concept text spans from free-text reports, and each clinical concept is aligned in its own dedicated visual subspace, turning concept stratification into direct, top-level alignment supervision. Instantiated on chest radiographs with a 19-concept schema over $203{,}602$ examinations from an internal multi-year archive, RadPRISM improved internal dataset zero-shot classification from $0.717$ (95% CI, $0.710-0.723$) to $0.868$ (95% CI, $0.863-0.872$) macro AUROC over a matched global-alignment baseline, performed on par with the purpose-built CARZero reference in external zero-shot classification while substantially outperforming it (up to 4.3-fold) in pointing-game visual grounding. In addition, a radiologist reader study demonstrated concept-stratified retrieval ability ($0.78$ macro retrieval correctness rate within rank 3), surfacing disentangled descriptive findings that report-level retrieval and fixed-label vocabularies cannot express. RadPRISM yields discriminative, spatially faithful, natively concept-stratified representations shaped by and transparently inspectable by clinicians.

Problem

• vision-language pretraining
• concept disentanglement
• radiology reports
• model interpretability
• visual grounding

Innovation

• concept-disentangled representations
• schema-stratified supervision
• visual grounding
• vision-language pretraining
• radiology report parsing