<p> Adversarial attacks aim to mislead trackers by injecting imperceptible perturbations into video frames in visual object tracking. Most existing methods generate perturbations across multiple frames, leading to high computational costs and increased detection risks. While some approaches target only the template frame but often depend on pre-trained perturbation generators, which involve high training costs and offer limited transferability across different tracker architectures. To address these challenges, we introduce an attack method that optimizes perturbations solely on the template frame. Specifically, we propose a diffusion model–based attack framework that eliminates the need for additional generators. By incorporating adversarial guidance during the diffusion process, the original template is transformed into a deceptive adversarial one. Additionally, we develop a defense strategy to restore original features. Experiments show that our attack framework significantly impairs tracker performance even with low attack frequency, causing a performance decline of up to 81%, while maintaining an SSIM of 0.927. The proposed defense demonstrates adaptability and recovery across various perturbations. Both frameworks are effective and practical.</p>