Unleashing the Potential of Deep Learning for Precise Nuclei Segmentation and Classification in H &E-Stained Whole Slide Images

Accurately identifying diverse cell types within and in the vicinity of the tumor matrix plays a crucial role in understanding the tumor microenvironment for cancer prognosis and scientific investigation. By automating the detection, segmentation, and classification of nuclei using Artificial Intelligence (AI), the burden on pathologists can be alleviated, minimizing errors caused by fatigue and subjectivity. In this study, we conducted a comprehensive evaluation of five pre-trained models: Patch DenseNet121, Patch EfficientNet-B3, Patch MobileNet, Patch VGG-16, Patch ResNet34, and a custom Patch Attention U-Net. These models offer an end-to-end solution for the automated segmentation and classification of nuclei from H &E-stained whole slide images of multiple organs (breast, kidney, lung, and prostate) using the MoNuSAC 2020 Challenge dataset. Performance was assessed using the F1-score and Intersection over Union (IoU) metrics. Among the models tested, Patch VGG-16 emerged as the top-performing model, achieving an impressive IoU of 85.48 and an F1-score of 0.88 on the validation dataset. The results of this study contribute to the advancement of AI-based techniques for the automated analysis of tumor microenvironments in diverse organ types, ultimately aiding in cancer diagnosis and treatment decision-making.