Chart recognition in document images is essential in digitizing and understanding document information. This paper proposes a novel method to classify the chart image by redefining class categories by composing multiple familiar and biased classes into larger, more representative clusters. Our approach involves a multi-step process wherein we amalgamate related classes to form composite classes, subsequently performing classification within these composed clusters. By iteratively refining classifications within each cluster, we effectively mitigate the effects of class imbalance and enhance the accuracy of predictions. Significantly, our method facilitates the improvement of classification accuracy for classes with similar features without necessitating additional data collection efforts. Our approach has been evaluated through empirical testing via a public dataset: UBPMC. The experimental results show the efficacy of our strategy in handling imbalanced datasets and improving model performance in scenarios where traditional techniques may fall short. The source code is available at github.com/lakiet1609/Chart-Classification .