<p>In engineering practices, the shear failure of rock joints often shows a nonorthogonal shear failure regard to the strike angle of the joints. The co-angle between shear direction and the orientation of rock joints is defined as the rotation angle. To investigate the effect of rotation angle on shear failure mechanism of saw-tooth joints, a series of numerical simulation on models with different rotation angles and undulation angles is carried out under different normal stresses. Peak shear strength calculation formula considering rotation angle was established and verified. The results show that the shape of shear stress-displacement curves can be divided into three types, slip curve, slip-peak type curve and peak type. Four different failure modes of rock joints with different undulation angles under nonorthogonal shear are observed, including slip, dilation, shear off and friction. The shear strength increases with the increase of the normal stress and undulation angle, and decreases with the rotation angle increase from 0° to 90°. Subsequently, a shear strength empirical formula considering different rotation angles, undulation angles and normal stresses is proposed. Comparative analysis shows that the formula behaves in a reliable state in calculating peak shear strength of joints under nonorthogonal shear in constant normal stress condition. Furthermore, the findings could contribute to the improvement of rock failure caused by joint failure in nonorthogonal mode.</p>