Music beat and downbeat tracking is a fundamental task in music information retrieval (MIR) and is widely used in many fields of practical and academic research, such as stage lighting arrangement, music structure analysis and music similarity detection. In this paper, we propose a hybrid network named XBeat that combines CNN and Transformer modules. The Feature Exchange Unit (FEU) is designed to merge local features and global representations at different scales. Moreover, We analyze the characteristics of the beat and downbeat tracking task. A continuous large mask for the input spectrogram would have a huge impact on the music beat tracking task, so we improved a simple spectrogram-based data augment method. At last, we demonstrate the improvement of our proposed model and data augmentation method through ablation experiments. The results indicate that, under similar model parameter quantities, XBeat has achieved a certain improvement compared to the widely used TCN in recent years.