Evaluation metrics are essential to compare the performance of algorithms. In particular, segmenting complex tubular structures, such as neuron dendrites and vascular networks, requires taking into account the object’s topology, calling for more specialized evaluation metrics than the traditional Dice score. In recent years, numerous topology-aware evaluation metrics have been proposed but were designed for specific applications and demonstrated using only a few selected examples, making it challenging to identify their limitations in different biomedical contexts. In this work, we introduce a quantitative method to analyze and compare segmentation evaluation metrics. We propose to classify segmentation errors into interpretable categories, such as “disconnections” and “radius dilation”. We generate a synthetic dataset of predicted segmentations by artificially introducing errors of specified types to ground-truth labels. This dataset allows us to measure and visualize the impact of each type of error on a given metric. We applied the proposed method to assess the strengths and weaknesses of eight SOTA evaluation metrics, guiding the selection of the most suitable metric for a target application. The dataset and code are made available to support the development and characterization of new metrics and losses: https://github.com/megdec/BenchmarkTopoSegMetrics/ .