Although Transformer-based methods have become the mainstream solution for single object tracking tasks, their lack of spatial inductive bias makes it difficult for them to effectively distinguish between targets and background in complex scenes. Moreover, relying on positional correlations between patches often causes the model to attend to local regions instead of the entire target, resulting in tracking box drift. To address these issues, we propose a coarse-to-fine target localization approach for object tracking, named Adaptive Pruning Cross-domain Fusion (AdaCF) Tracking. Firstly, The last-rank elimination attention selection strategy, called Routing Attention, is employed to adaptively prune background information that is dissimilar to the target features. This mechanism not only enhances the model’s ability to distinguish the target from the background by reducing background noise interference, but also significantly improves inference efficiency by eliminating redundant computations. Secondly, in the cross-domain fusion module, we first apply the Discrete Cosine Transform (DCT) to map spatial information into the frequency domain. Then, we implement joint spatial-frequency enhancement. This enables the model to further distinguish between the target and interfering information, such as accompanying objects, similar backgrounds, or look-alike objects. Thirdly, we introduce the Efficient Intersection over Union Loss (EIOU), which significantly accelerates model convergence and improves the localization accuracy of anchor boxes. The experimental results on the GOT-10k dataset demonstrate that our method significantly improves tracking performance, with AO increased by 3.3% and \(\text {SR}_{0.5}\) by 4.1%, validating its enhanced stability and adaptability in complex scenes.