<p>Object counting can be formulated as a density estimation task using point-annotated images. Although such labeling is cost-effective, trained models can be sensitive to annotation noise. In this paper, we propose a method called DUMLO (Distribution Uncertainty Matching for Loss Optimization) that defines a loss function between a ground-truth density map and a target density map by modeling uncertainty over an augmented set of points. DUMLO formulates the loss function as a coupling between two optimal transport problems, which involves an unknown density map defined over the augmented points. To solve the problem, we propose a new algorithm, called Trihorn, which jointly estimates the loss function and the density map of the augmentation set. The latter can be interpreted as a measure of the uncertainty associated with the annotations. We provide a theoretical analysis and show that the generalization error bound of the proposed loss is tight. We extensively evaluate our model on benchmark datasets from three real-world applications: pathology cell counting, crowd counting and Vehicle Images Datasets. Our results demonstrate that the proposed model achieves good performance in terms of Mean Absolute Error and is robust to annotation noise while exhibiting a fast convergence property.</p>