MATNet: a multi-attention transformer network for nuclei segmentation in thymoma histopathology images

Nuclei segmentation in thymoma histopathology images is essential for nuclei feature extraction and thymoma diagnosis. However, the variety, ambiguity, and overlapping of nuclei and the scarcity of available datasets challenge the nuclei segmentation tasks. This paper aims to develop a deep learning-based nuclei segmentation method to enhance the segmentation performance. We propose a multi-attention transformer network (MATNet) for thymoma nuclei segmentation. This network first uses convolution layers and transformer layers containing weighted position-sensitive self-attention to extract local and global information from small datasets. Then, it uses skip connection cross-attention to aggregate information. We also introduce a weighted loss function and an automated post-processing method to segment error-prone regions correctly, including touching nuclei. We conducted experiments on the thymoma histopathology image dataset we constructed. The proposed MATNet outperforms the other latest methods on all evaluation metrics, with accuracy, dice coefficient, ensemble dice coefficient, aggregated Jaccard index, and panoptic quality reaching 93.77%, 91.88%, 76.50%, 67.39%, and 67.93%, respectively. The proposed method accurately and automatically segments nuclei with excellent quantitative and visual results, especially for touching nuclei. This work contributes to the computer-assisted diagnosis of thymoma and is easily extended to other histopathology image segmentation tasks.