Automatic beat and downbeat tracking is an important research direction in the field of music information retrieval. This paper proposes a beat tracking algorithm based on multi-scale feature fusion and attention mechanism for the joint tracking of beat and downbeat. Firstly, we propose a convolution feature extraction layer based on multiscale feature fusion, which makes the model pay attention to different levels of music information and exchange musical instrument information with separated tracks. Then, based on the dilated self-attention, we introduce the dilated neighborhood attention module and the global attention module with multi-scale features. The former not only reduces the time complexity, but also realizes the information exchange of time instrument dimension characteristics, and improves the accuracy of beat detection; The latter can determine the global optimal beat sequence while fusing the time information of different scales, which improves the stability of beat detection. By comprehensively utilizing the information of different musical levels and a variety of attention mechanisms, our model can better perceive the global and local characteristics of beat. We performed experimental verification on four widely used datasets, including ballroom, Hainsworth, harmonic and Carnatic datasets. The experimental results show that, compared with the deep learning method in recent years, our proposed model shows better performance in beat tracking and downbeat tracking. Compared with baseline, the F-measure indexes of beat tracking and downbeat tracking on ballroom dataset are improved by 1.2% and 2.8% respectively.