RGBT tracking aims to leverage complementary RGB and thermal information for robust visual tracking. However, existing methods often fail when one modality becomes unreliable (e.g., due to low illumination or thermal crossover), largely because their fusion mechanisms are static or suboptimal. To address this, we propose WAE-Track, a novel tracker that tackles this challenge from two complementary perspectives: intra-modal detail enhancement and cross-modal adaptive fusion. First, we introduce a Wavelet-based Detail Enhancement Module (WDEM), which decomposes features in the frequency domain to explicitly preserve high-frequency details crucial for precise localization, effectively counteracting information loss in deep Vision Transformer backbones. Second, and more critically, we design a Cross-Modal Adaptive Enhancement (CMAE) module. It features a lightweight yet powerful Adaptive Gating Unit (AGU) that learns to dynamically modulate the interaction between RGB and thermal features. This enables our tracker to intelligently suppress corrupted information from an unreliable modality while amplifying features from the reliable one. Extensive experiments on three challenging benchmarks—GTOT, RGBT234, and the large-scale LasHeR—demonstrate that WAE-Track sets a new state-of-the-art. Notably, it achieves a remarkable 71.0% Precision Rate on LasHeR, significantly outperforming previous leading methods and validating the effectiveness of our proposed approach.