Traffic accidents are one of the top ten causes of death, with driver fatigue accounting for a significant proportion. Fatigue can lead to reduced attention and slower reaction times. Many professions require frequent nighttime driving, such as truck and taxi drivers, making them particularly a high risk group. Therefore, preventing accidents caused by driver fatigue, especially during nighttime, is a crucial task. Currently, there is limited research and datasets focused on fatigue driving in nighttime scenes. To address this gap we collect nighttime fatigue driving data using an infrared (IR) camera, and propose Unbalanced LocalCNNs for fatigue driving detection in this work. The network architecture can effectively direct the network’s attention to different regions based on specific actions caused by fatigue, and result in a 1.6% improvement in accuracy compared to the original models. Furthermore, an adversarial learning mechanism is introduced to enhance the network’s robustness, ensuring effective feature extraction in both day and night scenarios. Compared to models without adversarial learning, the overall accuracy is improved by 1.5%. The code is available at https://github.com/KaiChun-Tu/slow fastDrowsyDriver.