Unmanned Aerial Vehicles (UAVs) are growing increasingly popular, but they also pose severe security challenges. Object tracking is a vital technique for anti-UAV defenses. However, most of the existing tracking algorithms are constrained to utilizing either visible or infrared image data, rendering them ineffective at exploiting multi-modal data. Besides, these trackers rely heavily on target template matching for UAV localization. To tackle these issues, this paper presents a keypoints trajectory tracking approach combining visible and infrared data. Firstly, our solution employs a Dynamic Multi-modal Registration module to fix the misalignment between two modalities in the Anti-UAV dataset. Then the registered visible and infrared images are fused using a Layered Multi-modal Image Fusion network. Finally, with the Keypoint Trajectory Tracking module and Online Adaptive Calibration module, the UAV position can be predicted exactly on the fused data. Experimental results reveal a significant improvement in our method compared to state-of-the-art trackers, especially in challenging scenarios like out-of-view and tiny targets. The code and results are available at https://github.com/muqiu791/prcv .