Predicting stochastic pedestrian trajectories is inherently complex, which requires the integration of contextual information and the inherent uncertainty of human movement. Traditional generative models like GANs and CVAEs often fall short due to training instability and producing unnatural trajectories. Recent advancements have introduced Denoising Diffusion Probabilistic Models (DDPMs), which reduce prediction uncertainty but struggle to integrate past trajectory data effectively and are slow for real-time applications. We propose an innovative conditional diffusion model that enhances trajectory prediction by introducing a novel denoiser with a cross-attention mechanism to integrate past trajectories, ensuring alignment with historical behavior patterns and potential future changes. Additionally, we incorporate adaptive Fourier transforms to improve temporal analysis, capturing pedestrian movements across different time scales. To address slow inference speeds, we employ Denoising Diffusion Implicit Models (DDIM) for accelerated sampling. Rigorous benchmarking on ETH and UCY datasets demonstrates our approach achieves state-of-the-art performance in generating accurate trajectory distributions, surpassing existing methods in both efficiency and accuracy.