<p>In this paper, a bit-rock interaction model considering the surface morphology of the rotary drilling rig bit and the rock is established. Considering the complex shapes of the drill pipes and their intricate connections, a nonlinear multivariable model for the rotary drilling rig is developed. Based on this model, a stability prediction method for the rotary drilling rig is established. Numerical simulations and field experiments for drilling gravelly soil were conducted under various control parameters. The results indicate that the proposed dynamic model and stability prediction method for the rotary drilling rig are accurate. The stability of rotary drilling rig operation is inversely proportional to the strength of rock. As rock strength increases, the stable working area of the bit decreases. In addition, the drilling stability of rotary drilling rig is also affected by the change of extension states of the drill string. High-stiffness steel ropes help improve the stability of the drilling system. Within a specific range of torsional speeds, increasing the ground support stiffness helps to expand the stable range of the system.</p>