Heart rate estimation from face videos has proven to be a promising approach for non-invasive vital monitoring. However, noise and illumination variations can significantly impact the accuracy and reliability of heart rate prediction methods. This paper proposes a novel model that addresses these challenges by leveraging an attention mechanism followed by a transformer-based architecture. Our model extracts facial patches from video frames and integrates local and global context information across frames to capture meaningful representations. We carried out extensive experiments on three publicly available datasets, encompassing both intra-dataset and cross-dataset scenarios, to evaluate the performance and generalizability of our proposed method. The results demonstrate that our attention-based transformer model outperforms state-of-the-art techniques, exhibiting superior accuracy and robustness in heart rate prediction. Our model offers significant advancements in heart rate estimation from face videos by effectively reducing noise and handling illumination variations. The intra and cross-dataset validation experiments showcase the model's ability to generalise well across diverse datasets, underscoring its potential for real-world applications.