Charts are rich with data and ubiquitous in scientific research, which makes the ability to automatically extract structured data from them a research topic of great interest. An end-to-end pipeline that is capable of building datasets from raw chart data could generate significant value in both research and industry. The first step in automated chart content extraction is classification, but major challenges that impede progress for this task include limited datasets, class similarity, class imbalance, and overall chart heterogeneity. The CHART-Info 2024 dataset captures the diversity and challenges of this domain with a set of over 50,000 real-world chart images categorized into 15 different classes. Using this dataset, we studied different data augmentation and training techniques on existing deep learning architectures, such as vision transformers (ViTs) and self-distillation with no labels (DINO) models. Our experiments resulted in a final chart classification model that achieved a macro-averaged F1 score of 94.90%, outperforming the previous state-of-the-art on CHART-Info 2024 by 1.30%. Analysis of our results suggests that masking portions of each training image and overlaying gridlines as an augmentation enabled our models to learn better features and reduce overfitting. Also, we found that the class imbalance problem can be ameliorated by training the model with weighted random samples of the original dataset distribution that give higher priority to underrepresented classes.