<p>Rotate vector (RV) reducer is one of the crucial parts of industrial robots, which has the advantages of compact structure, strong load-bearing capacity, and high efficiency, and it is extensively applied in industrial production and mechanical manufacturing. The cycloidal mechanism, as the second-stage reduction component of the RV reducer, significantly influences the reducer’s performance. Therefore, it is necessary to conduct the research of meshing characteristics analysis for the cycloidal mechanism and explore the feasible methods to improve the performance of the mechanism. In this paper, the cycloidal mechanism of RV-40E reducer is selected as a research object. Firstly, the transmission principle and meshing pressure angle of the cycloidal mechanism were analyzed. Based on gear meshing principle and the Hertz contact theory, the contact stiffness equation and the torsional stiffness model of cycloidal mechanism were derived and established. Next, the effects of structural and modification parameters on the pressure angle and torsional stiffness were systematically analyzed. Then, taking the above parameters as variables and the pressure angle and torsional stiffness as objectives, the single- and multi-objective optimization designs for the cycloidal mechanism were carried out by applying the particle swarm optimization (PSO) algorithm. The mechanism parameters under various optimization cases are compared and discussed. Finally, the finite element simulations were performed for the cycloidal mechanisms before and after optimization for verifying the effectiveness of the stiffness model and optimization model. The results indicate that the proposed optimization method can significantly enhance the overall performance of the cycloidal mechanism.</p>