Despite advances in trajectory prediction, critical challenges persist in complex scenarios. On the one hand, existing methods struggle to capture intricate multi-agent interactions and long-term temporal dependencies; on the other hand, the high redundancy of traditional generative models in the computational process of gradually denoising makes it challenging to meet real-time requirements. To address these issues, we proposed TrafusionNet, an efficient trajectory prediction framework consisting of a perceptual modeling stage and an accelerated diffusion stage. In the perceptual modeling stage, a temporal-interaction perception module is designed to capture the multi-agent dynamic relationships through interactive cognition. In the accelerated diffusion stage, TrafusionNet samples directly from the latent trajectories generated by the perceptual modeling, skipping redundant denoising steps to optimize sampling efficiency and accuracy. In addition, by generating multiple future latent latent trajectories to address uncertainty, a balance between diversity and accuracy can be achieved. The experimental results on the NBA and ETH- UCY datasets showed that TrafusionNet outperformed existing methods in terms of error, model efficiency, and sampling time, achieving the most advanced predictive performance currently available.