TransPPG: two-stream transformer for remote heart rate estimate

Non-contact heart rate estimation using remote photoplethysmography (rPPG) has shown great potential in many applications and achieved creditable results in constrained scenarios. However, practical applications require results to be accurate even under complex environment. In this paper, we propose a novel video embedding method that embeds each facial video into a feature map referred to as Multi-scale Adaptive Spatial and Temporal Map with Overlap (MAST_Mop), which contains not only vital information but also surrounding information, which acts as the mirror to figure out the homogeneous perturbations imposed on foreground and background simultaneously, such as illumination instability. Correspondingly, we propose a two-stream Transformer to map the MAST_Mop into heart rate (HR), where one stream follows the pulse signal in the facial area while the other figures out the perturbation signal from the surrounding region to enable feature-level subtraction between the two channels. Due to the context-aware overall feature embedding, the proposed approach outperforms all current state-of-the-art methods on two public datasets MAHNOB-HCI and VIPL-HR.