Structural damage assessment has become a necessity in the modern era. Researchers are trying to come up with better and faster ways to regulate structures’ health since traditional nondestructive testing (NDT) methods such as visual inspection take longer time to carry out and it is sensitive to the user’s skill in operating the apparatuses. Various methods incorporating reverse algorithms have been explored with the help of machine learning; however, we found out that the use of pretrained convolutional neural network (CNN) in detecting damage has not been explored widely. In this study, an ensemble network of CNNs is built based on GoogLeNet architecture to test its capability in detecting structural damages in plates. There are two cases of plate structure being tested for the model: isotropic plate (metallic structure) and orthotropic plate (composite structure). The damage induced in those plates is simulated with a reduction in mechanical properties, that is, elastic modulus in isotropic case and multidirectional elastic modulus in orthotropic case. The models try to pinpoint the location parameters of the damage in the plate and to quantify the severity of the damage itself by getting input variables from the modal properties of the plates. From the individual models, the information is then gathered using an ensemble network which is expected to improve the overall accuracy. The results from the final model show good correlation between predicted parameters and the actual case with promising results for further research.