Vehicle load monitoring is a challenging problem in highway infrastructures non-intrusive monitoring taking the advantages of innovative computer vision techniques. Vehicle classification in an axle-based fine-grained level is a vital to perform further load identification of different vehicle types in accordance with similar operational characteristics. While fine-grained deep image classifiers have obtained promising results in a wide variety of classification applications, they also present significant drawbacks, particularly concerning minority accuracy and misclassification interpretability. Datasets of highway vehicle classes are intrinsically imbalanced, in which the overwhelming majority are small vehicles. However, volumes of truck traffic are the primary pavement design inputs as trucks dominate the highway deterioration. Moreover, various vehicle classes comprise limited types of vehicle component: axle, tractor (or power unit), and trailer. In this study, a fully convolution network (FCN)-based semantic segmentation model is proposed to obtain pixelwise classification of the three vehicle components from input images of individual vehicles. With camera calibration, the semantic output masks are sufficient to obtain accurate axle spacings and height estimation of vehicle body units. Then, the object measurements are utilized as decision nodes of a fine-grained vehicle classification decision tree. The proposed method was able to classify individual vehicles captured from a highway traffic flow into FHWA classes with a competitive accuracy.