Structural system identification is crucial for downstream research such as response prediction, reliability analysis and structural performance assessment. Challenges still remain for system identification of full-scale structures. When minimizing the difference between experimental and model-predicted data for identifying a structural system, experimental modal parameters could be used but information contained in original measured response data may be lost. Measured response data are difficult to use because it is almost impossible to measure excitations of a full-scale structure to be input in its system model for generating predicted responses. This work proposes to use power spectral density (PSD) data for system identification. The advantage is that original information in measured data can be used, and only a small number of parameters characterizing statistical properties of excitations are to be identified with structural parameters, instead of requiring measured excitations. One problem is that many large PSD matrices need to be processed. In view of the good performance of convolutional neural networks (CNNs) in dealing with large matrices, a CNN-based system identification method is proposed, building a mapping between PSD data and structural parameters. The governing equation of the structural system is enforced in training. By doing this, the structural dynamics characterized by the governing equation can be used to reduce the amount of required data and get better accuracy, while the hidden physics that is difficult to model can be learned by CNNs. Leveraging on the analytical properties of CNNs, auto differentiation is proposed to efficiently train CNNs and quantify the associated uncertainties. The proposed method was numerically verified, and the results show that it can identify the system that accurately predicts dynamic behaviors of the target structure.