Rail-mounted gantry (RMG) cranes are critical handling equipment. The problem of traveling gantry gnawing on rail affects equipment safety and operational efficiency, and also increases maintenance costs, presenting a long-standing technical challenge in the crane industry. This paper proposes a method for establishing a rigid-flexible coupling dynamic model to investigate the rail gnawing phenomenon of RMG cranes. Focusing on a 37 m span RMG crane, a rigid-flexible coupling dynamic model is developed using ADMAS software and ANSYS software. The influence of three factors of different gantry dynamic stiffness, different installation methods of the wheels, and different wheel tread forms on the rail gnawing is tested on the rigid-flexible coupling dynamic model. The results show that: 1) when the dynamic stiffness of the gantry in the direction of the trolley is reduced to less than 0.60 Hz, the rail gnawing is significantly increased, while when the dynamic stiffness is increased to around 0.65 Hz, the rail gnawing is significantly alleviated; 2) the overall inclination angle of the traveling gantry wheels is between 0.1 deg and 0.15 deg, which can effectively reduce rail gnawing; 3) the use of specific tread shapes for traveling gantry wheels, such as curved or M-shaped shapes, can significantly alleviates the phenomenon of rail gnawing. Therefore, adopting the method proposed in this paper during the RMG crane design stage can prevent the rail gnawing phenomenon.