A Deep Ensemble Approach for Lung Disease Classification in Chest X-Ray Across Data Distribution Shifts and Unseen Data Generalization

Advancement of deep learning algorithms and their application in medical imaging has demonstrated expert-level performance in recent times in connection with diagnosis of different illness of patients. However, their accuracy may drop in the real world as training dataset cannot cover all aspects of distribution. This problem is often encountered when deep neural networks are exposed to out-of-distribution (OOD) instances. This demands the need for employing appropriate technique to handle OOD instances as it is not practically feasible to retrain the models over and over again every time changes occur in distribution of data. In this study, a stack generalization deep ensemble learning approach is proposed for the classification of lung diseases in chest X-ray images. Specifically, this approach includes training a series of \({M}_{n}\) $M_n$ models on independent \({D}_{in}\) $D_{\mathit{in}}$ internal datasets, and then training an advanced meta-model \({M}_{mlp}\) $M_{\mathit{mlp}}$ with the benefits of combined generalization with maximum confidence. Finally, the method has been evaluated with subsets of in-distribution \({D}_{in,ext}\) $D_{in,ext}$ datasets as well as entirely new OOD dataset \({D}_{out}\) $D_{\mathit{out}}$ . The proposed method yields AUC of 0.86, 1.00, and 0.99 for \({D}_{in, test}\) $D_{in,test}$ test sets. For three entirely new target domain datasets, AUC for \({D}_{in,ext}\) $D_{in,ext}$ is 0.89, while AUC for \({D}_{out}\) $D_{\mathit{out}}$ are found to be 0.86, 0.81, 0.80, and 0.55. This paper highlights the effectiveness of the proposed method in generalization to data distribution shifts in 2D chest X-ray medical imaging data.