Rotary-wing UAV is an innovative solution that can replace manual labor to complete aerial photography and video, inspection and logistics work, solving the problems of safety risks and low efficiency of traditional operations. Geomagnetic sensors obtain heading information by measuring the direction and strength of the earth’s magnetic field, and a high-precision heading measurement system is the basis for rotary-wing UAV attitude control and localization. In this paper, a geomagnetic sensor’s own error and hard magnetic interference error correction method is introduced, and the sampling process is to turn and only turn the RUAV horizontally and vertically one turn each, and the fitting result is ellipsoid 9 parameters; A GPS assisted soft magnetic interference error detection method is also proposed to automatically adjust the geomagnetic heading correction weights. Validation tests were carried out in the independently developed RUAV, and the standard deviation of the corrected geomagnetic field vector strength was better than 0.01 Gauss, which can meet the needs of rotary-wing UAV hovering and other tasks, and has a wide range of popularization and application value.