Grey Level Texture Features for Segmentation of Chromogenic Dye RNAscope from Breast Cancer Tissue

Chromogenic RNAscope dye and haematoxylin staining of cancer tissue facilitates diagnosis of the cancer type and subsequent treatment, and fits well into existing pathology workflows. However, manual quantification of the RNAscope transcripts (dots), which signify gene expression, is prohibitively time consuming. In addition, there is a lack of verified supporting methods for quantification and analysis. This paper investigates the usefulness of gray level texture features for automatically segmenting and classifying the positions of RNAscope transcripts from breast cancer tissue. Feature analysis showed that a small set of gray level features, including Gray Level Dependence Matrix and Neighbouring Gray Tone Difference Matrix features, were well suited for the task. The automated method performed similarly to expert annotators at identifying the positions of RNAscope transcripts, with an \(F_1\) -score of 0.571 compared to the expert inter-rater \(F_1\) -score of 0.596. These results demonstrate the potential of gray level texture features for automated quantification of RNAscope in the pathology workflow.