Fatigue driving poses a significant threat to road traffic safety. Identifying a driver’s fatigue state is essential to improve driving safety. However, existing fatigue driving detection methods often involve a large number of parameters, making it difficult to deploy on resource-constrained devices. To address this challenge and maintain high accuracy while achieving a lightweight design, we propose GEP-SN, a lightweight two-stage framework for fatigue driving detection. In the first stage, the efficient multi-scale attention mechanism (EMA) is integrated into the Ghost module to propose the GE-Bottleneck, which replaces the inverted residual blocks in the original PFLD backbone. Meanwhile, a feature fusion module (FFM) is introduced before the fully connection layer, forming the optimized GE-PFLD architecture. In the second stage, the extracted facial keypoints and corresponding image frames are fed into a ShuffleNetV2 model for further feature extraction and fatigue state classification. Experimental results show that GE-PFLD outperforms the baseline models in both keypoint localization accuracy and inference speed with a small number of parameters. GEP-SN achieves an accuracy of 99.55% on YawDD dataset, validating its effectiveness in practical fatigue driving detection applications.