Purpose <p>In rotor systems, the non-smooth contact between rotor and stator may lead to wear and failure of components, and even suddenly induce backward whirl of rotor with large-amplitude oscillation to trigger dynamic instability of the entire unit. The energy transfer during contacts renders the motion inertia of stator and damping of system non-negligible. This work is devoted to gaining a deep insight of the essence underlying the rotor/stator rubbing and the resulting dynamic behaviors.</p> Method <p>This paper establishes a 4-DOF dynamic model to describe the dissipative rotor/stator rubbing system with motion inertia and structural damping. In modal analysis, a virtual damping is introduced to balance the energy dissipation of this non-conservative system. A neural network-based harmonic expansion method (NNHEM) is proposed to efficiently and accurately obtain the nonlinear modal characteristics of this non-conservative system. The energy transfers in rotor-dominant and stator-dominant motions during rub-impact are revealed respectively by employing modal damping and modal frequency.</p> Results and Conclusions <p>Compared to the regular HBM and its variants, the NNHEM eliminates the need for extensive symbolic manipulations and does not require complex frequency-domain transformations by combining discrete time-domain collocation points with the nonlinear approximation capability of neural networks. By this way, we found that the mode with forward whirl always shows positive modal damping across interested energy levels, indicating the stable operation of rotor. In contrast, the mode with backward whirl exhibits a range of negative modal damping within certain energy intervals. Under this mode, the energy from frictional forces is continuously introduced into the system and surpasses the energy dissipated by structural damping. This is manifested as friction-induced self-excited vibration in the response, ultimately evolving into dry friction backward whirl with a larger amplitude.</p>