As an important part of high-speed train health management, the ability of axle box bearing fault diagnosis technology in multi-task has been focused by researchers in recent years. At present, multi-task fault diagnosis methods based on deep learning are normally limited by data conditions, and many algorithm models training only rely on basic bench laboratory data. Moreover, the multi-task information representation ability of the model is limited. To solve these challenges, a multi-scale fault diagnosis model for multi-task is proposed. The model adopts dynamic simulation vibration signal data which simulates real vehicle operation to fully train multi-scale deep learning fault diagnosis model based on CNN architecture, and effectively diagnoses axle box bearing faults from two perspectives of fault degree and fault location. The experimental results show that this method performs well in multi-task diagnosis of axle box bearing fault.