Phenology refers to the seasonal life events and cycles of organisms, encompassing behaviors and physiological changes such as growth, flowering, fruiting, migration, reproduction, and dormancy in both plants and animals. It holds significant implications for agriculture, ecology, climate research, and natural resource management. Accurate counting of plant organs from herbarium specimens is crucial for inferring phenological indices, particularly when assessing the effects of climate change on plant growth and development. In phenological and plant growth analyses, localizing plant reproductive organs proves to be more informative than simple counting. This paper introduces an innovative end-to-end framework for localizing and counting plant organs. Our approach involves pretraining on a dataset of 700,000 images, followed by preprocessing through Coarse Point Refinement (CPR). The framework then directly predicts a set of point coordinates representing plant organs through CP2PNet and associates them with ground truths using the KL2 matching algorithm. This method improves the accuracy and utility of phenological studies by providing precise localization and counting of plant reproductive organs.