Autonomous driving systems are rapidly advancing, yet their robustness in diverse weather conditions remains a significant challenge. Adverse weather, such as rain, snow, sand, and fog, can severely impair the performance of vision-based algorithms. Collecting real-world data under these conditions is difficult due to high failure costs and safety risks, making traditional data collection and annotation impractical. To address this, we introduce Weather-Diff, a novel framework based on diffusion models for generating realistic adversarial weather conditions from clear weather images. Unlike traditional GAN-based methods, which can only conduct one-to-one domain translation, Weather-Diff can conduct adversarial weather generation to arbitrary domains given a target image for structure and semantic and reference images for style in a test-time optimization-based manner. The core techniques behind Weather-Diff include a Noise Prior Translation Block to mimic the reference style and a Denoising Trajectory Optimization Block to enhance semantic coherence and structural alignment with the target image. Additionally, an attention-based style injection mechanism and a dynamic guidance scale strategy ensure fine-grained control over the generated outputs. Quantitative evaluations demonstrate that Weather-Diff outperforms existing methods on metrics such as PSNR, SSIM, FID, and KID, setting a new standard for weather condition translation in autonomous driving contexts.