Machine learning based visible light image recognition of normal temperature metal fitting temperatures has achieved good laboratory results. However, in substation fields, diverse metal fitting materials and difficulty in collecting high-temperature fault images lead to data imbalance, reducing model accuracy. This study focuses on how the imbalanced data of three materials (copper, iron, and aluminum) affect the error in recognizing metal fitting temperature rise faults. Image libraries of these materials, collected at different angles, are established. The existing image database is defined as the baseline database, and rarely collected samples (images of 50–100 ℃) are defined as the update database, which is added to the baseline database each time. Two partitioning schemes simulate different data imbalance field scenarios. Under varying degrees of data imbalance, four machine learning algorithms (kNN, DT, GBRT, and RFR) predict temperatures of medium- and high-temperature images hard to collect in the field, and their impact on the model is analyzed. This study offers theoretical and experimental guidance for optimizing substation metal fitting temperature rise fault recognition models and for creating rare sample collection strategies for different field materials.