The damage is abundant in engineering structures that have been in service for a long time. Vibration-based nondestructive testing as a global method has been extensively used in structural damage identification. In this study, a multi-stage damage identification method combining 2D Singular Value Decomposition (SVD), Faster region-based convolutional neural network (Faster R-CNN) and particle swarm algorithm (PSO) in plate structures is presented. A semi-analytical dynamic model of elastically restrained plates for notched damages is proposed to evaluate damage effect on plate vibration performance. Based on the proposed model, the significant challenges in identifying location, type and severity of damage are solved. The fundamental aspects of multi-stage damage identification method in term of notch damage’s location, type and severity are assessed. First, two-dimensional SVD is presented to apply to the mode shape (MS) to detect damage location. It demonstrates that the proposed two-dimension SVD is superior in noise immunity. The types of notch damages are divided into five classifications. Secondly, the results of the SVD are used as the input to the Faster R-CNN, and the damage type can be identified by training the Faster R-CNN. Finally, the PSO was utilized to obtain the damage severity in the previously established natural frequency database (NFD). Numerical simulation results show that the proposed multi-stage damage detection method can effectively identify the notched damage in the plate.