<p>The dynamics of nonlinear systems often exhibit the coexistence of attractors, which is referred to as multistability or multistable dynamics. The final state of motion in a multistable system is closely related to the choice of initial conditions. However, in most cases, it is desirable to avoid multistable dynamics or to control the system to a specific state in complex environments. This paper studies a class of nonsmooth gear transmission systems and reveals a variety of multistable phenomena. We obtain the basin of attraction by an optimized cell mapping approach, and rich coexisting phenomena can be found, such as the coexistence of attractors with different periods and the coexistence of periodic and chaotic attractors. In the multistable parameter range of the system, higher-order approximate solutions of the coexisting attractors are obtained by the incremental harmonic balance method, which are compared with numerical solutions to validate the effectiveness, and its convergence of different harmonic orders is demonstrated by analyzing the residuals. Additionally, different control methods of multistability are applied to the multistable phenomena, and then, large-amplitude periodic and chaotic motions can be controlled to the desired motion, enabling the system to switch between different stable states. In addition, the advantages and disadvantages of each control method are analyzed, and the energy consumption characteristics of three control strategies is compared and analyzed through an energy consumption index. This study not only improves the stability of gear transmission systems, but also extends their acceptable life.</p>