Convolutional neural networks (CNNs) have achieved remarkable performance in various medical image segmentation tasks. However, the success of CNNs heavily relies on a large amount of accurately annotated training samples, which are challenging and expensive to obtain in medical image analysis. In clinical settings, the task of annotating lesions is gradually being performed by trained non-expert personnel, making data more accessible. Leveraging non-expert annotated data to extract reliable information for lesion segmentation, addressing the challenge of obtaining expert annotations for segmentation tasks by utilizing these labels. In this paper, we propose an architecture called DiffTri-Network for medical image segmentation tasks using non-expert annotations, aiming to alleviate the challenges of data unavailability. Specifically, our architecture employs network models of different depths that fuse their respective extracted reliable features, guiding the network’s learning process and updating their parameters respectively. These three networks achieve collaborative optimization through this co-learning approach. In order to simulate non-expert annotated medical images in real-life scenarios, we mixed the benign and malignant tumor labels in the BUSI dataset and applied dilation, scaling, and jitter operations. To ensure the objectivity of the data, the data generated from these three operations were randomly partitioned. We compare our method on this dataset with fully supervised segmentation methods. Experimental results show that our method is capable of extracting more image features from such data and achieves excellent segmentation performance.