Fault detection in power distribution networks is crucial for the stable operation of the grid and the continuous supply of power. However, faults in distribution networks exhibit significant randomness in their occurrence and duration. Due to their complexity and non-stationarity, existing methods frequently encounter issues with false positives and missed detections. To address these challenges, we introduce a novel fault detection model based on the Transformer architecture, incorporating a newly proposed Sliding Window Energy Attention (SWEA) mechanism and a time-frequency feature extraction structure. Within this model, each encoder’s time-domain channel utilizes sliding window techniques to extract energy features and calculate attention scores, while the frequency-domain channel employs Fast Fourier Transform (FFT) to extract the fault spectrum. The final fault detection results are outputted by a Multilayer Perceptron (MLP). Experimental validation on a real distribution network fault dataset demonstrates the significant advantages in detection performance of the proposed model over selected baseline models.