Improving the Accuracy of Predictive Models in Imbalanced Lung Cancer Data

The accurate diagnosis of lung cancer using predictive modeling presents significant challenges, primarily due to the imbalanced nature of clinical datasets where certain outcomes are underrepresented. This study addresses the critical impact of class imbalance on the predictive accuracy of machine learning models applied to lung cancer diagnosis. We evaluated several popular classification algorithms, including K-Nearest Neighbors (KNN), Support Vector Machines (SVM), Random Forests, and Deep Learning models, across original and various enhanced datasets. Our methodology involved preprocessing the data to handle missing values and applying several techniques to balance the classes effectively. These techniques included manual oversampling, undersampling, and synthetic oversampling methods. The manual oversampling allowed us to duplicate instances of the minority classes, while undersampling reduced the instances of the majority class. Synthetic oversampling, using a method like the Adaptive Synthetic Sampling (ADASYN), generated new synthetic instances for the minority classes. This combined approach allowed us to enhance the representation of minority classes and improve the generalizability of our models.The performance of each model was assessed using accuracy metrics and Receiver Operating Characteristic (ROC) curves across both dataset conditions. Results indicated that SVM, Random Forest, and Deep Learning models, when trained on balanced data, demonstrated significant improvements in accuracy and ROC-AUC scores compared to training on the original imbalanced dataset. Specifically, the Random Forest and Deep Learning models showed a notable increase in performance, highlighting the effectiveness of ensemble and deep learning methods in dealing with class imbalances. This study confirms that addressing class imbalance through a combination of manual and synthetic oversampling techniques can substantially improve the accuracy of predictive models in lung cancer diagnosis. These findings advocate for the integration of these techniques in preprocessing steps for clinical data analysis, potentially leading to more reliable and equitable healthcare outcomes.