Residual stresses and their distribution over the depth of subsurface layers are one of the most important characteristics of the forming process of various products. This characteristic essentially determines the service properties and durability of these products. In this study, a methodology to determine the residual stresses generated in the subsurface layers of gear teeth as a result of different finishing machining of these gears is proposed. Experimental studies of residual stress distribution in gear teeth subjected to gear honing with tools on metal and elastic bonds, as well as combined versions of these tools have been carried out. Residual stresses after grinding of gears with a globoid wheel and after machining of gear teeth by the method of single division at point contact of the machined gear with tools equipped using cutting elements with high-density boron nitride modification are also determined. A model experiment for determining residual stresses at various points of tooth profile height is proposed, and a numerical model for modeling the residual stresses in workpiece subsurface layers formed as a result of orthogonal cutting process is developed. The deviations between the results of experimental studies, model experiment and numerical simulation of residual stresses show the promising application of the developed methods.