The accessory gearbox transmission system is a critical component for the service of aero-engine. The internal gear system of aero-engine experiences high rotational speeds and heavy loads during the service, resulting in complex dynamic excitations from multiple gear pairs. The system resonance, consequently, severely challenged the reliability and performance of the internal gear system. This study developed a finite element model of the accessory gearbox transmission system and gearbox casing. Transient finite element simulations were employed to capture the natural frequencies of the system, meshing stress contour, and dynamic meshing force evolutions under extreme operating conditions. It is identified from the prediction that the gear pairs exhibited bias loading, significant meshing force fluctuations, and a complex internal excitation spectrum. System resonance is observed with the excitation of five modes, necessitating frequency tuning to mitigate resonance effects. Sensitivity analysis of the system’s modal characteristics reveals that the natural frequency is more sensitive to spoke thickness than shaft thickness and rim diameter. Reducing the spoke thickness of gears Z1 and Z6 by 2 mm effectively avoids resonance frequencies, resulting in a 61.56% and 31.83% reduction in meshing force fluctuations for the corresponding gear pairs. The simulation-based gear system structure optimization improved the meshing performance and the reliability.