Evaluating trustworthiness of decision tree learning algorithms based on equivalence checking

Learning algorithms and their implementations are used as black-boxes to produce decision trees, e.g., for realizing critical classification tasks. A low confidence in (the learning ability of) the algorithms increases the mistrust of the produced decision trees, which leads to costly test and validation activities and to the waste of the learning time in case the decision trees are likely to be faulty due to the inability to learn. Methods for evaluating trustworthiness of the algorithms are needed especially when the testing of the learned decision trees is also challenging. We propose a novel oracle-centered approach to the evaluation. It consists of generating deterministic or noise-free datasets from reference trees playing the role of oracles, producing learned trees with existing (implementations of) learning algorithms, and determining the degree of equivalence (DOE) of the learned trees by comparing them with the oracles. We evaluate (six implementations of) five decision tree learning algorithms based on the proposed approach.