Fair and Accurate Skin Disease Image Classification by Alignment with Clinical Labels

Deep learning models have achieved great success in automating skin lesion diagnosis. However, the ethnic disparity in these models’ predictions needs to be addressed before deployment of these models. We introduce a novel approach: PatchAlign, to enhance skin condition image classification accuracy and fairness through alignment with clinical text representations of skin conditions. PatchAlign uses Graph Optimal Transport (GOT) Loss as a regularizer to perform cross-domain alignment. The representations thus obtained are robust and generalize well across skin tones, even with limited training samples. To reduce the effect of noise/artifacts in clinical dermatology images, we propose a learnable Masked Graph Optimal Transport for cross-domain alignment that further improves the fairness metrics. We compare our model to the SOTA model (FairDisCo) on two skin lesion datasets with different skin types: Fitzpatrick17k and Diverse Dermatology Images (DDI). Our proposed approach, PatchAlign, enhances the accuracy of skin condition image classification by 2.8% (in-domain) and 6.2% (out-domain) on Fitzpatrick17k and 4.2% (in-domain) on DDI compared to FairDisCo. In addition, it consistently improves the fairness of true positive rates across skin tones in all of our experiments.