<p>The railway vehicle is a high-dimensional and self-excited system characterized by complex modal coupling effects. Abnormal vibrations of carbody chattering, induced by system instability, occasionally occur on rail lines, significantly deteriorating passenger comfort and threatening operational safety. This study investigates the coupling features between carbody flexibility and hunting motion. A rigid-flexible coupling model that considers carbody elastic vibration is established. The analysis of system dynamics features comprises two main aspects. First, the continuous modal tracking technology is applied to describe and explain this mode-coupling instability phenomenon. The mode veering phenomenon is observed when the hunting frequency approaches the natural frequency of carbody’s diamond mode, and a strong coupling effect between carbody’s diamond mode and hunting mode is clearly identified. Second, the dynamic behavior is analyzed in detail with the help of bifurcation diagrams, amplitude-frequency spectrums and phase plane diagrams. A new parameterization scheme is proposed to describe the strong nonlinearity of wheel-rail contact geometry. The result shows that a large initial and negative slope of the equivalent conicity function causes a frequency jump phenomenon in bifurcation, indicating strong modal-coupling between carbody’s diamond mode and hunting motion. Additionally, the high-order flexible modes of carbody also resonate with multiple harmonics of hunting motion. The results suggest that suspension optimization and structural improvement are promising methods to mitigate the carbody chattering phenomenon.</p>