Stochastic trajectory prediction plays a pivotal role in various applications, such as autonomous driving and robotic navigation. As one of the most mainstream approaches, diffusion-based stochastic trajectory prediction algorithms have achieved remarkable progress. However, they overlook the fine-grained temporal analysis of the trajectories, and the large number of diffusion steps makes it challenging to ensure real-time performance. To address these limitations, this paper proposes a Dynamic Diffusion-based stochastic trajectory prediction with Time-frequency Analysis (DDTA), which focuses on optimizing both the trajectory encoder and diffusion model. DDTA consists of two core components: the Multi-scale Time-frequency Analysis Module (MTAM) and the Dynamic Diffusion Module (DDM). (1) The MTAM, comprising a wavelet-based decomposer and a pyramid-based aggregator, is designed to analyze the instantaneous characteristics and behavioral patterns of temporal features and generate trajectory features through multi-scale fusion. (2) The DDM leverages trajectory features as a diffusion initialization prior and dynamically adjusts the number of diffusion steps to significantly reduce computational overhead. By combining the strengths of MTAM and DDM, DDTA enhances algorithm performance while effectively improving inference efficiency. Extensive comparative, ablation, visualization, and efficiency experiments conducted on three datasets (ETH-UCY, NBA, and SDD) demonstrate the effectiveness and superiority of the proposed method.