In this paper, we propose a novel Trajectory-Conditioned Diffusion (TCD) model for adaptive image restoration under diverse and compound weather degradations. Traditional methods mainly rely on explicit weather labels to guide the restoration trajectory, leading to inefficient diffusion processes and struggling with compound weather degradations. To address this issue, we introduce optimal transport theory into the diffusion process that adaptively modulates the reverse diffusion trajectory without explicit weather labels. Specifically, TCD consists of two stages. The optimal transport module first aligns diverse corrupted inputs with clean target images in the latent space through cost-effective restoration trajectories. The diffusion process disentangles residual diffusion and noise diffusion and then reconstructs fidelity images from aligned latent features. Extensive experiments demonstrate that our method outperforms existing approaches on both synthetic and real-world benchmarks, particularly in handling complex weather scenarios.