Positron emission tomography (PET) is widely recognized as the most sensitive molecular imaging modality, enabling the in vivo visualization of molecular pathways. Despite its exceptional utility, concerns about ionizing radiation exposure have limited its broader application. A recent breakthrough in total-body PET imaging addresses this limitation by significantly increasing geometric coverage and sensitivity. This innovation reduces radiation exposure to levels comparable to the dose received during a transatlantic flight, achieved through advanced computational techniques. To accelerate progress in this field, we have curated a benchmark dataset specifically designed for developing ultra-low dose PET imaging methodologies. This dataset was pivotal in the Ultra-Low Dose PET Imaging Challenge held in 2022, 2023, and 2024. The challenge aimed to foster innovative computational algorithms capable of recovering high-quality imaging from low-dose scans acquired on total-body PET systems. The dataset includes both standard-dose and simulated low-dose total-body PET images from 1,447 patients. These were acquired using Siemens Biograph Vision Quadra PET/CT and United Imaging uExplorer PET/CT scanners. In addition, we have developed a customized evaluation system to assess the performance of algorithms in recovering image quality from low-dose scans. This paper provides a comprehensive description of the benchmark dataset and evaluation framework, aimed at driving future advancements in ultra-low dose PET imaging. The dataset is available at https://udpet-challenge.github.io , subject to the completion of a signed Data Transfer Agreement.