On the Capability of Classification Trees and Random Forests to Estimate Probabilities

With the rising popularity of artificial intelligence, machine learning algorithms are being considered for an increasing number of problems. For binary classification, most algorithms can provide an estimate of the probability that an event will occur, but the statistical properties thereof are often unknown. After reviewing convergence results for classification trees and random forests in the literature, we discuss how some methods could be negatively impacted by poor probability estimates. We design an extensive Monte Carlo simulation inspired by nine datasets to evaluate the ability of different algorithms to estimate probabilities. We find that while trees and forests may perform better at ranking, their ability to estimate probabilities rarely exceeds that of logistic regression, even when the logistic regression is misspecified.