Machine learning is indispensable for biomedical data modeling and classification. Tasks involving large, high-dimensional datasets are nevertheless computationally intensive and approximation methods are often sought to scale down the volume of raw data or model size without compromising substantial information embedded within the data. However, previous approximation methods have yielded mixed results and have yet to establish a clear framework linking feature selection and model sparsification. In this paper, we present an information-theoretic approach for cancer classification by addressing two prominent questions in data model approximation: how to identify a minimal set of critical features in cancer microarray data and how to design sparse neural networks that are effective and efficient for cancer classification. Our study highlights a key connection between these two challenges. In particular, we introduce a mutual information (MI)-based method to select a highly informative subset of genes from extensive microarray gene expression data. Each selected subset of genes, up to two orders of magnitude smaller than the original gene set, demonstrates superior performance in cancer classification compared to the full dataset. Additionally, the MI-based method enables the design of sparsified neural networks that consistently maintain or even improve classification performance compared to fully connected networks. Our test results reveal that sparsified networks selectively retain connections to the critical genes identified by the MI-based filtering method, effectively ignoring contributions from irrelevant genes.