Music Emotion Recognition (MER) is a subfield of Music Information Retrieval (MIR). With the advancement of deep learning, computer-based MER has gained growing research interest, yet it faces challenges due to the subjectivity of music emotions: establishing links between complex music encodings and emotions is difficult, and labeled audio-emotion data is scarce. To address these issues, this study proposes an attention-based multi-scale MER model. Most existing models ignore low-level emotional features; thus, a multi-scale parallel branch structure is adopted to enable the model to fully learn both low-level and high-level music features, which are critical for MER and help capture more comprehensive emotional information to boost recognition accuracy. Considering that music emotions involve spatial and temporal features, a polar self-attention mechanism is integrated to focus on fine-grained spectrogram information and enhance feature extraction. Additionally, Temporal Convolutional Network (TCN) is used for music feature extraction—its dilated and causal convolutions achieve a larger receptive field with fewer parameters, facilitating the capture of music’s historical information. Experimental results on the PMEmo dataset show promising performance: for Arousal, the Root Mean Squared Error (RMSE) is 0.1104 and R2 is 0.6724; for Valence, RMSE is 0.1155 and R2 is 0.5022.