This study explores the decision-making processes of machine learning models for text classification problems. Using a fake news dataset as a test case, the study compares neural networks and machine learning approaches to fake news detection. This text-based problem has a feature space of 12,569 dimensions, and the necessity of the full dimensionality of the data is explored. In addition to developing effective classifiers, this study aims to investigate neural network interpretability by applying an explainable AI framework to extract human-understandable rules from trained models. The rule extraction process, taking a pedagogical approach, investigates the decision making of models. A Boolean function based model was developed, and the extent to which this rule-based system over a reduced feature set is successful is evaluated.