Existing 3D medical image segmentation methods are often constrained by a fixed set of predefined classes or by reliance on manually defined prompts such as bounding boxes and scribbles, which are often labor-intensive and prone to ambiguity. To address these limitations, we present a framework for 3D medical image segmentation across diverse modalities guided solely by free-text descriptions of target anatomies or diseases. Our solution is built on a multi-component architecture that integrates efficient feature encoding via decomposed 3D convolutions and self-attention, multi-scale text-visual alignment, and a SAM-inspired mask decoder with iterative refinement. The model is further conditioned through a prompt encoder that transforms language and intermediate visual cues into spatially aligned embeddings. To train and evaluate our model, we used a large-scale dataset of over 200,000 3D image-mask pairs spanning CT, MRI, PET, ultrasound, and microscopy. Our method achieved an average Dice of 0.6091 and F1_50 score of 0.1131 on the open validation set, outperforming baselines such as CAT (Dice 0.5316, F1_50 0.1935) and SAT (Dice 0.5573, F1_50 0.0956). It showed strong generalization across modalities, with particularly high performance on ultrasound (Dice 0.8337) and CT (Dice 0.6707). These results confirm the feasibility of free-text-guided 3D segmentation and establish our approach as a strong foundation model for general-purpose medical image segmentation. Our code is publicly available at: https://github.com/mirthAI/Text3DSAM/ .

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Text3DSAM: Text-Guided 3D Medical Image Segmentation Using SAM-Inspired Architecture

  • Yu Xin,
  • Gorkem Can Ates,
  • Wei Shao

摘要

Existing 3D medical image segmentation methods are often constrained by a fixed set of predefined classes or by reliance on manually defined prompts such as bounding boxes and scribbles, which are often labor-intensive and prone to ambiguity. To address these limitations, we present a framework for 3D medical image segmentation across diverse modalities guided solely by free-text descriptions of target anatomies or diseases. Our solution is built on a multi-component architecture that integrates efficient feature encoding via decomposed 3D convolutions and self-attention, multi-scale text-visual alignment, and a SAM-inspired mask decoder with iterative refinement. The model is further conditioned through a prompt encoder that transforms language and intermediate visual cues into spatially aligned embeddings. To train and evaluate our model, we used a large-scale dataset of over 200,000 3D image-mask pairs spanning CT, MRI, PET, ultrasound, and microscopy. Our method achieved an average Dice of 0.6091 and F1_50 score of 0.1131 on the open validation set, outperforming baselines such as CAT (Dice 0.5316, F1_50 0.1935) and SAT (Dice 0.5573, F1_50 0.0956). It showed strong generalization across modalities, with particularly high performance on ultrasound (Dice 0.8337) and CT (Dice 0.6707). These results confirm the feasibility of free-text-guided 3D segmentation and establish our approach as a strong foundation model for general-purpose medical image segmentation. Our code is publicly available at: https://github.com/mirthAI/Text3DSAM/ .