Recurrent breathing cessation during sleep, known as sleep apnea (SA), is a widespread sleep disorder. if not treated on time, may lead to serious health complications. Many studies use deep learning methods to detect sleep apnea automatically from single-lead electrocardiogram signals. However, these studies have overlooked the impact of temporal features on detection performance. We propose a multi-granularity temporal sleep network (MTSN) to detect sleep apnea and address this challenge. The MTSN model consists mainly of three modules. The Intra-segment temporal feature module extracts fine-grained temporal features within segments through multi scale convolution kernels. The Inter-segment temporal feature module uses a temporal convolution network to learn long term temporal features between segments. The deep attention refinement module enhances important features through variants of channel attention. We conducted experiments on the public dataset to demonstrate the model’s effectiveness. Our proposed MTSN model achieved an accuracy of 91.95% and a sensitivity and specificity of 88.15% and 94.30%, respectively, with an F1 score of 89.34%.