<p>The probe center position is pivotal in enhancing the on-machine measurement accuracy of gear grinding machine. Presently, the measurement of involute gear tooth profiles is primarily conducted offline, with probe calibration typically relying on Standard sphere. Notably, the mandrel can serve not only to calibrate the central position of the probe but also to verify the static measurement accuracy of the gear grinding machine. In this paper, a novel, simple, and efficient on-machine calibration method using the mandrel for probe center position is proposed. By leveraging the principle of relative methods and homogeneous coordinate transformation theory, a new mathematical model is established to quantify the <i>Y-axis</i> and <i>Z-axis</i> deviations of the probe center position. The effect law of the probe center position on tooth profile deviation is analyzed through self-developed virtual prototype, guiding practical machine measurements. Additionally, the minimum circle coverage algorithm based on random increment is employed to compensate for probe center position errors. Experimental results indicate that, according to international ISO standard, the on-machine measurement accuracy of both left and right tooth flanks reaches Level 6 before error compensation. After implementing the proposed calibration and compensation method, the measurement accuracy of both tooth flanks improves to Level 4, which aligns with the results obtained from the Gleason 350GMS offline measurement system. This study offers valuable insights into enhancing the on-machine measurement accuracy of gear grinding machine.</p>