Precision and Recall Reject Curves

For some classification scenarios, it is desirable to use only those classification instances a trained model associates with a high certainty. To evaluate such high-certainty instances, previous work proposed accuracy-reject curves (ARCs). Reject curves evaluate and compare the performance of different certainty measures over a range of thresholds for accepting or rejecting classifications. However, in some scenarios, e. g., data with imbalanced class distributions, the accuracy may not be a suitable evaluation metric and measures like precision or recall may be preferable. We therefore propose reject curves that evaluate precision and recall, the recall-reject curve and the precision-reject curve. Using prototype-based classifiers, we first validate the proposed curves on artificial benchmark data against the ARC as a baseline. We then show on benchmarks and medical, real-world data with class imbalances that the proposed precision- and recall-curves yield more accurate insights into classifier performance than ARCs.