Ovarian tumors (OT) are one of the most common tumors in the female reproductive system, and are often classified into two categories: benign and malignant. B-mode ultrasound (BMUS) and contrast-enhanced ultrasound (CEUS) are commonly used as conventional imaging techniques for ovarian tumor screening; however, image interpretation is time-consuming. In this study, we construct a multi-task multi-modal ultrasound image dataset. A multi-task model based on a pre-trained deep learning architecture is trained and applied to predict and select key clinical parameters, as well as to classify benign and ovarian tumors. Finally, blood flow signal, morphology, septation, and solid component are selected as key clinical parameters related to the classification of benign and malignant ovarian tumors. The evaluation metrics for classifying benign and malignant ovarian tumors are as follows: accuracy 0.731, precision 0.692, recall 0.643, and F1-score 0.653. The method based on deep learning and medical imaging has the potential to assist in accelerating the initial screening of benign and malignant ovarian tumors and generating image-to-text inspection reports.