This paper aims to investigate the prediction of the nonlinear shear model of composite materials using an enhanced neural network system. By modifying the backpropagation equation, the loss function is defined using the data measured in the experiment as the system output, it facilitates learning the shear nonlinear constitutive relationship of composite materials directly from measured data. Additionally, in the backpropagation stage, genetic algorithms are employed instead of backpropagation algorithms to improve the prediction accuracy of the shear nonlinear relationship of composite laminate plates. A comparison with the Adam algorithm under equivalent conditions confirms the feasibility of using the enhanced neural network system combined with genetic algorithms. This study provides new insights for accurately and efficiently predicting the shear nonlinear relationship of composite laminate plates.