With the tremendous advances made by large modern astronomical detectors and telescopes, the depth and range of observations of the sky by these devices is expanding, allowing for an enormous amount of imaging data to be collected. These data contain many radio galaxies, but due to their huge size, manual search for classification is not feasible. We thus study an automatic classifier for radio galaxies based on deep learning and adopt a new classification scheme for radio galaxy classification. Considering that radio galaxy classification is based on morphological features and needs to focus on feature scale information, we design a U-shaped multi-scale feature fusion network to achieve the fusion of deep and shallow feature information; and add an attention mechanism to allow the model to focus on information-rich features. We also use Fine-tune strategy of migration learning in the training process to speed up the convergence of the network model. The experimental results show that our model can achieve higher accuracy classification performance in the case of six types of classification of radio galaxies.