Siamese network based trackers have shown remarkable success in unmanned aerial vehicle (UAV) visual tracking due to their promising balance of accuracy and speed. However, in tracking tasks, it is common practice to pre-train the backbone networks on large-scale image recognition datasets. This requirement poses a limitation on our ability to apply or design lightweight backbone networks. To address this issue, we propose a simple yet effective Siamese network framework based on cross-head knowledge distillation, named SiamXKD. Specifically, the intermediate features of the student are passed to the tracking head of the teacher, allowing the student to inherit the knowledge of the teacher while mimicking the teacher’s predictions. Additionally, we integrate the correlation filter into the proposed framework, empowering the Siamese network with the ability to update online and enhancing its robustness. Extensive experiments on four well-known UAV visual tracking benchmarks demonstrate the superiority of SiamXKD, with a speed of about 200 FPS running on a GPU.