<p>The contact characteristics of the rough tooth surface during the meshing process are significantly affected by the lubrication state. The coupling effect of tooth surface roughness and lubrication on meshing characteristics of planetary gear is studied. An improved three-dimensional (3D) anisotropic tooth surface roughness fractal model is proposed based on the experimental parameters. Considering asperity contact and elastohydrodynamic lubrication (EHL), the contact load and flexibility deformation of the tooth surface are derived, and the deformation compatibility equation of the 3D loaded tooth contact analysis (3D-LTCA) method is improved. The asperity of the tooth surface changes the system from EHL to mixed lubrication and reduces the stiffness of the oil film. Compared with the sun-planet gear, the asperity has a greater effect on the meshing characteristics of the ring-planet gear. Compared with the proposed method, the comprehensive stiffness obtained by the traditional calculation method considering the lubrication effect is smaller, especially for the ring-planet gear. Compared with roughness, speed and viscosity, the meshing characteristics of planetary gears are most sensitive to torque.</p>