Open-Ended CT Volume Segmentation with Weak Supervision from Language

2026-07-28

AI Summary

This work addresses the high annotation cost and scarcity of strongly supervised data in open-vocabulary segmentation of CT images by proposing a novel approach that integrates voxel-level strong supervision with slice-level weak language supervision extracted from radiology reports. For the first time, lesion descriptions at the slice level from scan–report pairs are leveraged as weak supervisory signals and jointly trained with strong annotations to guide a SAM3-based text-prompted segmentation model. This strategy substantially reduces reliance on fully annotated data while enhancing model generalization. On the ReXGroundingCT dataset, the method achieves a relative Dice score improvement of 22% using only 250 fully annotated CT scans, and an 8% gain when scaled to 1,000 cases, demonstrating its effectiveness in leveraging limited annotations through multimodal supervision.

Abstract

We introduce a method for training a text-conditioned segmentation model for CT scans, which combines voxel-level supervision with coarse but scalable slice-level supervision from reports. We extract, from a large database of scan-report pairs, descriptions of findings with indices of slices where those findings occur. We then finetune a general-purpose 2D image segmentation model, SAM3, with standard segmentation losses from strongly labeled data and with a slice-level classification loss from the extracted weak supervision. Our results on the ReXGroundingCT dataset illustrate that this strategy improves the segmentation dice score: from an 8% relative gain when there are 1000 fully labeled volumes to 22% when there are 250 fully labeled volumes.

Problem

• CT volume segmentation
• weak supervision
• language guidance
• open-ended segmentation
• slice-level annotation

Innovation

• weak supervision
• text-conditioned segmentation
• CT volume segmentation
• slice-level annotation
• SAM3