It is a major challenge to understand the spatio-temporal interactions of driving scenarios in accident prediction tasks of intelligent vehicle systems. Given that the gaze information of experienced drivers during the driving process involves complex spatio-temporal interactions, this information can serve as human attention to provide guidance for the training of accident prediction models. However, most existing studies use the human attention information as one of the multitask labels and do not leverage multi-scale spatial channel information of human attention for supervision. To address this gap, we propose a Multi-Scale Adaptive Network (MSAN) guided by human attention for traffic accident prediction. This network efficiently learns multi-scale spatial channel information from human attention composed of driver’s eye movement information. Additionally, we introduce a novel strategy for the recursive transmission of time information to mitigate the impact of varying previous results on current model inference. After experiment, our model has been proven to excel over state-of-the-art approaches on two benchmark datasets, which serves as a practical solution to bolster the safety of intelligent vehicles.