<p>During geothermal and petroleum drilling, the drillstring undergoes complex coupled vibrations. Accurate prediction of its dynamic response is critical for drilling optimization, failure risk reduction, and efficiency improvement. This study establishes a nonlinear coupled-vibration model for rotating drillstrings using Rayleigh beam theory and the energy method. A displacement–load boundary conversion approach for the drill bit is proposed, which solves the problem of rotational-speed fluctuations during the stick phase of the drill bit in numerical simulations. The proposed model is validated against field and experimental data. Furthermore, based on bit displacement compatibility, a general expression for the displacement boundary of transient bit whirling is innovatively established, which accurately characterizes the inclined whirling phenomenon of the drill bit during actual drilling. The transient inclined whirling state of the drill bit is inversely deduced based on the established boundary conditions and wear morphology of the roller-cone bit obtained from geothermal drilling sites. The transient dynamic responses of the drillstring under high- and low-energy whirling states are analyzed, and the lateral vibrations of the drillstring are proved to be more severe under the high-energy whirling state, which tends to induce higher dynamic stress. This study expands the modeling methods for the motion and force states of the drill bit, providing a referential research approach for the analysis of similar complex bit-boundary conditions such as whirling and directional drilling. Furthermore, it lays a solid theoretical foundation for revealing the internal incentives of drillstring failure in the fields of geothermal and petroleum drilling.</p>