<p>The transfer matrix method (TMM) is an effective tool for analyzing the vibration characteristics of mechanical systems, but the nonlinear functions of eigenvalue contained in the transfer matrices limit the use of advanced matrix eigenvalue computation techniques, such as subspace iteration, block Lanczos iteration and others. To overcome this difficulty, this paper proposes a modified Newton method and an accelerated modified Newton method to determine the system eigenvalues. Firstly, the Jacobi’s formula is used to obtain the derivative of the characteristic determinant, and a numerical deflation is introduced to prevent the subsequent iterations from converging to previously obtained eigenvalues; then, the Newton iteration formula is improved to determine all eigenvalues. Secondly, to ensure the second-order convergence of the iterations for multiple eigenvalues, the Steffensen method is introduced to accelerate the iterations. Finally, the proposed methods are demonstrated through the modal analysis of a damped rotor-bearing system and an undamped structural system. The two proposed methods have higher computational efficiency and reliability than the direct bisection method with sub-interval search, recursive eigenvalue search algorithm and Lund’s method. So a more efficient tool is provided for determining eigenvalues of mechanical systems based on TMM.</p>