This study investigates the churn lubrication mechanism in gear transmission systems with nanofluids. Various performance parameters of the prepared nanofluid lubrication oil are measured. Based on the VOF multiphase flow model and overset mesh method, a rotating fluid domain computational fluid dynamics numerical calculation model considering the oil–gas two-phase flow is proposed. The high-speed nanofluid lubrication experiment platform is established to track churn phenomena and identify the oil volume fraction on the gear surface. The comparison of simulation results with experimental data confirms the calculation accuracy in depicting oil flow and churn lubrication. Result shows that the interaction of droplets with the gear surface includes splash, rebound, adhesion, and diffusion. The oil is slowly drawn out of the oil tank and carried along with the teeth, causing churning. As it progresses toward the meshing zone, it gets squeezed out from the narrowing tooth gap. The spin effect is primarily determined by the tip diameter rather than the rotational speed. This study serves as a foundational exploration for future nanofluid lubrication and heat dissipation, offering insights into preventing oil film rupture, tooth surface wear, gluing and other failure mechanisms.