Sparse Concept Channels in Frozen 3D CT Vision Encoders

2026-07-23

AI Summary

This work addresses the limited understanding of how frozen 3D medical vision encoders represent clinical findings, particularly which channels encode specific radiological concepts and their spatial locations. The authors propose Concept Channel Probing (CCP), a training-free method that, for the first time, reveals clinical concepts are encoded by only about ten sparse channels. The approach demonstrates cross-architecture generalizability across multiple 3D vision–language models. Experimental results show that CCP significantly outperforms CT-CHAT in clinical F1 score (0.549 vs. 0.184) and BLEU (0.483 vs. 0.373), while achieving a 22-fold reduction in inference latency. Further channel ablation studies confirm the specificity of the identified concept-channel mappings.

Abstract

Large vision-language models are becoming increasingly dominant in 3D medical image interpretation, but we rarely know <i>which</i> internal units encode clinical findings or <i>where</i> that information lives in the representation. We first study this on a 3D chest vision-language model (Pillar-0) by probing its frozen vision embeddings. We show that (i) each radiological finding is encoded by a <i>sparse</i> set of ~10 vision-encoder channels that match full-feature classification performance and far exceed a zero-shot text prompting; (ii) turning off the channels tied to one finding, that finding's score collapses while unrelated labels stay stable; and (iii) the same sparse probe <i>replicates</i> on an architecturally unrelated 3D abdominal VLM (Merlin) suggesting a general property of frozen medical encoders. Our training-free concept channel probe (CCP) method, paired with a corpus-derived report template, outperforms published CT-CHAT on clinical efficacy and NLG metrics (F1 0.549 vs. 0.184; BLEU 0.483 vs. 0.373) at 22x lower latency. Our results provide a clear, reproducible characterization of how frozen medical encoders represent findings, demonstrating direct applicability across models.

Problem

• 3D medical image interpretation
• vision-language models
• clinical findings
• frozen vision encoders
• concept representation

Innovation

• sparse concept channels
• frozen vision encoder
• training-free probing
• 3D medical vision-language models
• clinical interpretability