Self-supervision drives representational convergence in medical foundation models more than clinical supervision

2026-07-22

AI Summary

This study investigates whether representations of medical foundation models converge, what drives such convergence, and their clinical utility—questions that remain unclear due to the fragility of existing similarity metrics. Through controlled experiments holding data, architecture, and scale constant, the authors train encoders with diverse objectives and replicate observed effects in synthetic models, systematically evaluating representational alignment across 18 image and 7 text encoders. They demonstrate for the first time that self-supervised objectives—not clinical supervision or model scale—are the primary driver of representational convergence in medical imaging. However, this convergence remains confined within modalities and fails to align with radiologists’ judgments. Notably, self-supervised encoders achieve 40.4% representational alignment on chest X-rays, substantially outperforming label-supervised (21.1%) and vision–language models (3.3%), with linear probes retaining approximately 85% performance when transferred across models.

Abstract

Medical image encoders from different groups are increasingly treated as interchangeable, on the assumption that scale and clinical supervision concentrate their representations onto a shared structure. Whether this convergence is real, what produces it, and whether it is clinically usable are untested, and the similarity measures behind such claims are fragile. We present a controlled dissection across 18 image and 7 text encoders, all open-weight and run locally, spanning 7M to 27B parameters and five imaging modalities, including 650,982 chest radiographs from six datasets. To isolate cause, we train encoders that vary only the objective under fixed data, architecture, and scale, and reproduce the effect in a synthetic model. Convergence is modest but above a random floor, driven by the self-supervised objective, not clinical supervision: matched self-supervised encoders aligned most (40.4% on chest radiography), with label-supervised (21.1%) and image-text (3.3%) far lower, and did not grow with size (Spearman 0.302, p=0.223) or capability. It is within-modality, does not reach clinical language, and does not reproduce how radiologists judge case similarity. Yet a linear classifier transfers across encoders and to five held-out hospitals, retaining about 85% of within-encoder performance. Convergence in medical imaging is therefore set by the pretraining objective, not inherited from scale or clinical supervision. Interoperability is accordingly something to design for through that objective, and to validate where the shared geometry is weakest, across patient subgroups and against clinical judgment.

Problem

• representational convergence
• medical foundation models
• self-supervision
• clinical supervision
• encoder interoperability

Innovation

• self-supervision
• representational convergence
• medical foundation models
• clinical supervision
• interoperability