<p>As a weak link in engineering structures, the shear mechanical properties of rock joints are crucial for evaluating the stability of rock masses. To investigate the mechanism of shear stress oscillation in regular toothed joints and its relationship with specimen size and material parameters, this study conducted direct shear tests on five wave angles and six dimensions using RDS-200 under different normal stresses. The results show that (1) The peak shear strength of regular tooth-shaped rock-like joints shows a positive linear relationship with normal stress and undulating angle. For fixed-size specimens, increasing the undulating angle enhances both peak strength and pre-peak stiffness. When the sample size increases, the peak strength of the joints with the same undulating angle initially increases and then decreases, while the pre-peak stiffness generally decreases. (2) Shear stress oscillations predominantly occur under low normal stress or small undulating angles. Larger undulating angles reduce the oscillation climbing ratio, interval length, and frequency. (3) Increasing specimen size amplifies the oscillation climbing ratio, interval length, and average amplitude but lowers frequency, while higher normal stresses decrease both the oscillation climbing ratio and the number of oscillations but increase amplitude. The results provide a reference for optimizing the design and stability of rock structure.</p>