Rich and discriminative semantic representation has become increasingly critical for robust visual object tracking. However, most trackers rely solely on the high-level semantic features of the final layer of the backbone network, neglecting the abundant fine-grained information present in lower layers. This limited semantic scope hinders the expressiveness and completeness of the target representation. To overcome this limitation, we propose SDFTrack, a novel and lightweight image-level tracking framework that effectively integrates both shallow and deep semantic features from the backbone, without any additional temporal information interaction. Specifically, we introduce a semantic information fusion module, which consists of a cross-attention layer followed by a self-attention layer. In this module, deep semantic features interact with shallow ones via a cross-attention layer to achieve coarse-grained semantic enhancement, while the self-attention layer further refines the fused representation to supply detailed target information dynamically. By seamlessly embedding this fusion module into the backbone architecture, SDFTrack maintains semantic alignment while significantly enriching the feature representation of the target. Extensive experiments conducted on several challenging benchmarks demonstrate that SDFTrack consistently outperforms existing methods with real-time inference speed. In particular, the base model achieves an average overlap of 79.2% on GOT-10k, setting a new performance record in one-shot. The code and models are available at https://github.com/xiaomengxin123/SDFTrack .