<p>Remote Photoplethysmography (rPPG), an emerging non-contact solution for measuring physiological signals by capturing subtle skin color variations from facial videos, has attracted considerable attention for its potential in many applications. While recent deep learning-based approaches primarily leverage CNNs or Transformers to map videos to rPPG signals, they often fail to fully exploit the intrinsic temporal periodicity of rPPG. Moreover, thanks to recent advances in the diffusion model and the inherent periodic distribution of rPPG, remote physiological measurement can also benefit from the diffusion model in estimating rPPG with high quality. In this work, we propose a MultiScale Fusing-based diffusion-based architecture, namely MSPhys, to effectively capture the multiscale temporal contextual and periodic rPPG clues for precise rPPG estimation. At the core of MSPhys, the <i>multiscale temporal fusing block</i> serves as the key denoising module, where input features are first decomposed into multiscale temporal representations via a down-sampling mechanism and subsequently refined through a learnable up-sampling process to integrate multiscale information. Additionally, we introduce a hierarchical loss function to constrain multiscale temporal consistency between the predicted and ground-truth rPPG signals, further improving estimation accuracy. Extensive experiments on four challenging physiological benchmark datasets demonstrate that MSPhys achieves state-of-the-art performance, significantly outperforming existing approaches in remote physiological measurement.</p>