Magnetic encoders have the advantages of small size, resistance to harsh environments, and strong anti-interference ability, and are widely used in the industrial field. In order to improve the resolution and accuracy of the magnetic encoder, this paper starts from the working principle of the magnetic encoder and the working principle of the magnetic sensor. Through detailed theoretical analysis and simulation verification of the output error of the magnetic encoder, after determining the source of the error, a combined sensor and a differential operational amplifier circuit are used on the hardware instead of the integrated high-speed magnetic angle encoder chip. The software uses the quadrature phase-locked loop method to solve the angle of the output signal after amplitude normalization and Kalman filtering algorithm compensation. The test results show that the output signal after error compensation is close to the ideal waveform. Compared with the uncorrected magnetic encoder, the maximum angle error is 0.211°, and the maximum angle error after angle solution is 0.142°, with an absolute accuracy of 12bit.