Automated Segmentation of Prostate in Male Pelvis Region Using Contextual Layer Neural Network

This paper presents a novel deep learning framework, N2, for the automated contouring of the prostate in male pelvic CT scans, addressing the limitations of existing methods in handling anatomical variability and ensuring temporal consistency. Building upon the widely adopted Swin Transformer-based U-Net architecture, our model integrates anatomical context by incorporating features from adjacent segmentation masks into its training process. We utilized the publicly available Prostate-Anatomical-Edge-Cases collection from The Cancer Imaging Archive (TCIA), which includes a diverse set of cases with anatomical variations, to rigorously train and evaluate the model. The N2 model incorporates historical segmentation masks through a dedicated Context Encoder and a Multi-head Cross-Attention Fusion mechanism, which enriches spatial representations with temporal cues and improves robustness to inter-frame variations. Our experiments demonstrate that the N2 model, particularly when trained with a Tversky loss function and a random temporal context gap, significantly outperforms the baseline Swin-UNet model. The best performance was achieved with Tversky loss parameters \(\alpha = 0.7\) , \(\beta = 0.7\) , resulting in a Dice Similarity Coefficient (DSC) of 0.978, which is a substantial improvement over the baseline model’s DSC of 0.890. These results highlight the critical role of integrating contextual information in deep learning models for accurate and robust medical image segmentation, particularly for challenging anatomical cases in radiation therapy planning.