Prompt-based object counting refers to estimating the number of object correspondence to a selected category based on the text description provided by the user. Current state-of-the-art methods estimate object counts by summing the values in the predicted density map, not caring about the distribution of object locations. This is reflected by their loss function, mainly MSE loss, a loss function that focuses solely on quantity. This leads to the model overestimating the count of object class due to certain factors like overlapping, occlusion, or object with the trait of self-similarity. To address this, we propose OptiCount, a framework using Optimal transport plan to measure the difference between the density map and ground truth for training. Furthermore, we introduce a density-refinement module that validates the number of objects counted to avoid overcounting. This module significantly reduces the counting error of the model, making it more robust to various challenges. Experiments on the FSC147 dataset show that OptiCount outperforms state-of-the-art methods in terms of Mean Absolute Error (MAE), demonstrating its effectiveness in counting task.