<p>Sperm head morphology has been identified as a characteristic that can be used to predict a male’s semen quality. Here, harnessing the close relationship considering sperm head shape to quality and morphology, we propose a joint learning model for sperm head segmentation and morphological category prediction. In the model, the sperm category prediction and the ellipticity, calculated by using the segmented sperm head profile, are used to synthesize the morphology to which the sperm belongs. In traditional clinical testing, fertility experts analyze sperm morphology by 2D images of sperm samples, which cannot represent the whole character of their quality and morphological category. To overcome the problem that single-angle 2D images cannot accurately identify sperm morphology, we use a deep-learning-based tracking and detection system to dynamically acquire sperm images with multiple frames and angles and then use the multi-frame and multi-angle time-series images of sperm to determine sperm morphology based on the multi-task model proposed in this study. Performing better than 3D sperm reconstruction and traditional computer-assisted sperm assessment systems, this approach enables end-to-end analysis of viable spermatozoa, requiring minimal computing power and utilizing equipment already available in most embryology laboratories.</p> Graphical Abstract <p></p>