Thermal infrared (TIR) images generally exhibit low detail and contrast, making it difficult for conventional feature extraction models to identify distinctive target characteristics. As a result, trackers often struggle with interference from visually similar objects and are prone to tracking drift. To overcome these issues, we propose a novel saliency-guided Siamese network tracker based on key fine-grained feature information. Specifically, we introduce a fine-grained feature parallel learning convolutional block featuring a dual-stream architecture with convolutional kernels of different sizes. This structure captures crucial global features from shallow layers, enhances feature diversity, and reduces the loss of fine-grained information that commonly occurs with residual connections. We further design a multi-layer fine-grained feature fusion module that utilizes bilinear matrix multiplication to effectively integrate features from both deep and shallow layers. In addition, we propose a Siamese residual refinement block, which uses residual learning to correct errors in saliency map predictions. Supported by deep supervision, this refinement process progressively enhances prediction accuracy by applying supervision at each recursive step. Lastly, we introduce a saliency loss function that constrains the saliency predictions, guiding the network to concentrate on highly discriminative fine-grained features. Extensive experiments demonstrate that our proposed tracker achieves the highest precision and success rates on the PTB-TIR and LSOTB-TIR benchmarks.