A Two-Stage Coarse-to-Fine Detection Model Integrating YOLO11 and VGG16 for Cell Subtyping in High-Content Imaging

In toxicology research, accurate detection of cell subtypes is closely related to understanding the mechanisms of cell toxicity and providing early warnings for potential toxic responses. Traditional cell detection methods rely on microscopy and manual analysis, which are time-consuming and prone to subjective biases. In this paper, we develop a two-stage coarse-to-fine detection model integrating YOLO11 and VGG16 for cell subtyping in high-content imaging (HCI). In the coarse detection stage, YOLO11 is fine-tuned to efficiently identify different cell subtypes, including mononuclear (MoN), binuclear (BN), multinuclear (MuN), apoptosis (APOP), and necrosis (NEC) cells. In the fine detection stage, transfer learning is applied to the VGG16 model to further classify BN cells into subcellular types, including normal BN cells and abnormal BN cells with micronuclei (MN), nuclear buds (NB), and/or nucleoplasmic bridges (NPB), respectively. Experiments were conducted on 445 HCI images captured from the in vitro cytokinesis-block micronucleus (CBMN) assay. Our model outperforms others in detecting these five fundamental cell subtypes, with particularly impressive results in detecting BN cells, achieving a macro precision of 0.81, a recall rate of 0.89, and F1 score of 0.85. VGG16 shows strong performance in the fine detection stage with a macro precision of 0.83, recall of 0.76, and F1 score of 0.79. These results outperform other comparative deep learning models, underscoring the significant potential of our model for practical applications in automated detection of subcellular morphological motifs of HCI.