ALICE: Combining Feature Selection and Inter-Rater Agreeability for Machine Learning Insights

This paper presents a new Python library called Automated Learning for Insightful Comparison and Evaluation (ALICE), which merges conventional feature selection and the concept of inter-rater agreeability in a user-friendly manner to seek insights into black box Machine Learning models. The framework is proposed following an overview of the key concepts of interpretability in ML, and aims to fill an existing research gap by measuring agreeability between data-driven black box models and parametric white-boxes when the input feature set is iteratively manipulated. Initial experiments on a benchmark customer churn dataset for feature selection and predictive modeling suggest that Deep Neural Network Classifiers with a sigmoid head have more consistent and higher agreement with a simple Logistic Regression than a Random Forest Classifier. The full source code for the framework and the experiment notebooks can be found at: https://github.com/anasashb/aliceHU .