Deep Learning (DL) has become a dominant approach in malware detection, often favored for its ability to capture complex patterns. However, this study highlights a critical issue: simple non-deep learning models, such as One-Rule and AdaBoost, perform surprisingly well on several state-of-the-art malware detection datasets, achieving results comparable to those of more complex models, including LightGBM and DL approaches like FastAI’s Tabular Learner. Rather than suggesting that simple models are inherently superior, our findings indicate that a small subset of features often dominates the classification process, allowing models to achieve high accuracy with only a small fraction of the feature space. This behavior appears to be closely tied to the limited variability in some feature values across academic datasets. Across the eleven datasets that we analyze, classification scores reach above 99% and the performance gap between simple and complex models is only 5.6% in absolute terms and 6% in relative terms. We provide evidence that complex models are often overvalued in academic malware detection benchmarks and demonstrate the value of simple baselines such as the 1R model and AdaBoost. We also present examples of dominant features that bias classification across datasets, with an in-depth analysis of EMBER features and their impact on classification. Finally, we release cleaned, reusable datasets to support further research.