Data Entropy-Based Imbalanced Learning

All the time the skewness of observations is thought as the reason of poor classification performance, especially the bias in classification performance among classes in machine learning. However, our recent study challenges this notion. We argue that the bias of classification performance comes from the imbalance of information of classes rather than just that of observations. To reflect the information imbalance of classes, we propose an indicator data entropy that captures the randomness within classes. A dataset with balanced and higher data entropies across its classes is more likely to exhibit improved classification performance. Furthermore, we propose another indicator data mutual information that quantifies the similarity between classes. Higher values indicates that the models can leverage learning from classes to enhance learning capacity. Therefore, reducing the difference in data entropy between classes and enhancing data mutual information concurrently is advantageous for classification. Our experiments, conducted across four models SVM, CNN, Transformer (including its variants ViT), and DNN, on datasets CIFAR-10, Airline Satisfaction, Smoking Body Signal and Liver Cirrhosis, validate the efficacy of our proposed indicators. Through rebalancing the data entropy distribution among classes and increasing the data entropy within classes as well as the data mutual information in the Liver Cirrhosis dataset using resampling techniques, we observe classification enhancements measured in d-index across four models.