<p>The growing scale of wind turbines makes the blades more and more flexible, and the geometric nonlinear problems become prominent for wind turbine dynamic analysis. In this paper, a nonlinear model of a large wind turbine system based on the continuum mathematical approach is proposed. The flexible blades and tower are modeled as Euler–Bernoulli beams with considering cubic nonlinear terms. The derived model takes into account of the blade geometry nonlinear relationship and the effect of blade-tower coupling. It is validated by comparing the modal analysis with the published results. Further nonlinear dynamic analysis is carried out to analyze the effect of geometrical nonlinearity on the response of the wind turbine system, and the dynamic characteristics of the wind turbine system under different excitation amplitudes and rotational frequencies are discussed. It is found that the hardening effect can clearly checked when nonlinear effect is considered. The variation of rotational frequency of the turbine may lead to the occurrence of internal resonance phenomenon when there is an integer multiplicity relationship between the modes. When significant internal resonance occurs sudden change of response amplitude of blade can be induced, which requires careful treatment for design.</p>