With the increasingly severe trend of social aging, the detection measures for preventing the elderly from falling have become more and more urgent. And there are also more and more detection methods for the elderly with the development of deep learning, and the accuracy is getting higher and higher. Against this backdrop, we propose an innovative architecture combined with an experimental procedure suitable for the gait of the elderly, it classifies and detects the abnormal gait of the elderly based on the data obtained from the IMU (Inertial Measurement Unit) and Kinect. This architecture can integrate the information from the two experimental devices, addressing the problems of insufficient classification accuracy and inadequate information in the past single-modal gait information. Through ablation experiments, we have determined that the multi-modal gait anomaly detection method based on our architecture is more effective than the single-modal detection method. At the same time, when using the same multi-modal data, our architecture has made significant improvements compared with the traditional methods. In addition, this architecture can be adapted to the data from devices other than the IMU and Kinect. Before using it, it only needs to align the sequential data in time. query