<p>This paper undertakes a systematic investigation of the medical image segmentation benchmark datasets, which play a crucial role in the notable progress of medical image segmentation task. The datasets serve as the foundational infrastructure comparable to a backbone that supports and drives the development of medical image segmentation. Consequently, examination of these datasets emerges as a critical topic in research. In order to address the current lack of a systematic summary and thorough analysis of benchmark datasets for medical image segmentation, and to gain insights into their current status and future trends, this survey consolidates and categorizes the fundamental aspects of these benchmark datasets from five perspectives: (1) organ/tissue segmentation datasets; (2) lesion region segmentation datasets; (3) multi-organ/tissue segmentation datasets; (4) multi-lesion region segmentation datasets; (5) multi-organ/tissue and lesion region segmentation datasets. The survey elucidates the prevailing challenges and identifies potential avenues for future investigation. Additionally, a comprehensive review of the existing available dataset resources is also provided, including statistics from 140 datasets, covering seven core anatomical regions and over 60 subcategories of tissues/lesion types. Data from five segmentation task scenarios is incorporated into the dataset statistics. We aim to map out the full panorama of medical image segmentation datasets, serving as a comprehensive reference for researchers in this field and contributing to future studies. Related resources are available at: <a href="https://github.com/CVLife/Awesome-medical-segmentation-datasets">https://github.com/CVLife/Awesome-medical-segmentation-datasets</a>.</p>