<p>This study introduces a machine tool rotary axis error identification method that considers the constraints of the R-test theoretical model. First, a general modeling approach for the R-test was developed based on coordinate transformation theory, and a theoretical model was established for a four-sensor R-test. Then, theoretical constraints were considered, and multi-position measurement data were used across the entire measurement range to identify installation parameters. This approach improved the prediction accuracy of the transformation matrix while ensuring compliance with the theoretical model constraints. Furthermore, the effects of the proposed method, the least squares method, and the direct use of design values on the accuracy of the transformation matrix were analyzed through experiments. Finally, the geometric errors of the machine tool rotary axes were identified, and the effectiveness of the proposed method was verified by comparing experimental values with predicted values. This study not only presented a theoretical analysis method in addition to least squares identification but also providing a novel approach for identifying geometric errors in the machine tool’s rotary axes. While both the proposed method and the least squares method demonstrated high prediction accuracy, the proposed method proved more effective for small errors in machine tools and satisfied theoretical constraints.</p>