Taking class imbalance into account in open set recognition evaluation
摘要
In recent years, deep neural networks have been employed increasingly often, which correlates with them receiving growing user trust. However, such systems cannot identify samples from unknown classes and often induce an incorrect decision with high confidence. This is aimed to be solved by open set recognition methods. The presented work looks at the evaluation protocols of existing approaches in the field. A particular focus is being placed on the impact of class imbalance, especially in the dichotomy between known and unknown samples, which is a rarely considered factor in the experimental environment. The work analyzes current evaluation strategies—regarding dataset construction and metric selection—noting that the class imbalance can significantly impact the obtained results. We analyze the effect of using the popular baseline metrics (accuracy, balanced accuracy, and F1-score) for method quality assessment and introduce a protocol extension to four recognition quality measures that can be built upon those baselines. The analysis of base measures revealed that the choice of baseline metric could significantly impact the computed criterion values when the class imbalance of the recognized problem appears. The proposed experimental environment was used in an example experiment on commonly used computer vision datasets. As an outcome of problem analysis, we present a set of guidelines for evaluating open set recognition methods.