When Can Test-Time Adaptation Help Zero-Shot CT Vision-Language Models?

2026-07-16

AI Summary

This work addresses the challenge of unreliable multi-label anomaly prediction by zero-shot 3D CT vision-language models under distribution shift, a setting where existing test-time adaptation methods struggle due to the unsupervised and multi-label nature of the task. The paper formally defines and tackles this problem for the first time, introducing CARVE, a cardinality-aware framework. Through diagnostic analysis, it identifies conditions under which adaptation succeeds and integrates cardinality estimation, a top-k optimization objective, and a gradient-free multi-view scoring mechanism to enable efficient adaptation while preserving anomaly co-occurrence structures. Extensive experiments across diverse CT classification tasks demonstrate that CARVE consistently achieves significant performance gains, validating its effectiveness when the base model possesses discriminative capability.

Abstract

3D CT vision-language models (VLMs) classify abnormalities from text prompts in a zero-shot manner, enabling cross-institution deployment where labels are scarce and clinical tasks shift faster than supervised models can be retrained. A real CT scan, however, typically contains several co-occurring abnormalities, and the reliability of zero-shot multi-label prediction under distribution shift remains poorly understood. Test-time adaptation (TTA) updates a model on unlabeled target scans without source data or target annotations, yet existing TTA methods target multi-class softmax prediction on natural images or 2D medical segmentation, and none addresses unsupervised multi-label adaptation for zero-shot 3D CT VLMs. We study when TTA helps zero-shot 3D CT VLMs. A controlled diagnostic analysis shows that TTA is conditional: the volumetric input must preserve the encoder's depth structure, and the base representation must transfer to the target cohort, with depth reduction alone lowering internal AUROC by more than 0.12. We then focus on the regime where the base model already separates present from absent abnormalities. We introduce CARVE (Cardinality-Aware Retained-View Entropy), the first TTA method for this setting. CARVE estimates a sample-specific positive-label cardinality $\hat{k}$, optimizes a top-$\hat{k}$ objective to preserve co-occurring abnormalities, and performs memory-efficient multi-view adaptation by scoring weak 3D views without gradients before updating on a retained subset. Across contrastive CT-CLIP and anatomy-aware fVLM, CARVE provides the most consistent improvements across multi-label, three-class, and binary CT tasks when the base model is already discriminative. These results establish multi-label TTA for zero-shot 3D CT VLMs as a distinct problem and CARVE as a cardinality-aware solution.

Problem

• test-time adaptation
• zero-shot learning
• 3D CT
• vision-language models
• multi-label classification

Innovation

• Test-Time Adaptation
• Zero-Shot Learning
• 3D CT Vision-Language Models
• Multi-Label Classification
• Cardinality-Aware Adaptation