In the field of computer vision, Vision Transformers (ViTs) have demonstrated superior performance across a variety of downstream tasks. However, ViTs tend to lose a significant amount of fine-grained features (e.g., texture details) in synthetic aperture radar (SAR) ship detection due to the difficulty in capturing high-frequency (HF) information. Furthermore, the minuscule size of some ship targets in SAR imagery intensifies the challenge for ViTs in detecting small targets. To address these issues, we construct a multi-scale ViT fusing wavelet with HF enhancement (WHFE-ViT), making it retain more high-frequency information and enhance its detection ability for small ships. First, we adopt discrete wavelet transform (DWT) as down-sampling operation to reduce the computational cost and generate multi-scale features to accommodate ship targets of various sizes. Then, we design a HF enhancement path to enhance the texture features of ships. Finally, a feature fusion path is proposed to fuse time-domain and frequency-domain information, compensating for the lack of small ship features. After sufficient experimental verification, we validate the effectiveness of our model on HRSID dataset, with detection accuracy exceeding classical Convolutional Neural Networks (CNNs) and ViTs.