Our paper proposes a hybrid deep learning framework for segmenting ovarian cysts using advanced deep learning techniques and an improved U-Net architecture. This proposed model uses hybrid deep learning techniques to refine feature extraction and classification, in addition to leveraging U-Net’s strengths for capturing fine details in medical images. Extensive experiments demonstrate that the model outperforms traditional U-Net’s when it comes to segmenting ovarian cyst regions. In comparison with well-known methods like machine learning (97.2%), convolutional neural networks (96.6%), deep learning neural networks (95.7%), and support vector machines (95.2%), the proposed network’s accuracy was remarkable at 98.1%. Based on the results, both detection accuracy and loss reduction are enhanced by the network as it iterates over successive iterations. Based on this improved performance, it is obvious that the proposed model could provide more reliable and accurate ovarian cyst detection in real-world medical diagnostic applications.