Malaria remains a significant global health burden, and its timely diagnosis is critical for effective treatment and management. This research explores a hybrid approach combining CNN as a feature extractor with the XGBoost classifier for automated malaria cell image classification. The dataset used for this study comprises 27,558 images of malaria-infected and uninfected blood smear cells. Due to computational constraints, two reduced versions of the dataset were created: one containing 500 images and the other 1001 images, evenly split between infected and uninfected categories. Our approach leverages CNN’s ability to extract complex, high-level features from images and the efficiency of XGBoost in classification tasks. The model was trained over 30 epochs using a batch size of 64, with training and validation data evaluated across both datasets. The smaller dataset achieved an accuracy of 81.99% with an AUC of 0.916, while the larger dataset achieved a significantly higher accuracy of 94.65% and an AUC of 0.978. A hybrid model, combining CNN with XGBoost, further improved accuracy to 98.25%, with an AUC of 0.999, demonstrating the efficacy of this approach. Performance metrics, including accuracy, loss, and ROC curves, were used to evaluate the model. The analysis of training and validation accuracy and loss plots showed that the model converged well after 10–15 epochs, achieving near-perfect generalization on the test set. This research demonstrates that a CNN-XGBoost hybrid model can provide an accurate and efficient solution for malaria detection, even with reduced dataset sizes.