In recent years, non-contact physiological signal estimation based on remote photoplethysmography (rPPG) has gained increasing attention as a research hotspot. However, due to the subtle variations in facial skin color and the susceptibility to disturbances such as posture, illumination, and motion, improving the accuracy and robustness of estimation models remains a significant challenge. To address this, we propose an end-to-end dual-branch network architecture, named Multi-Task Spatial-Temporal Fusion Network (MT-STFuse). The model consists of a Temporal Fusion Branch and a Spatial Fusion Branch, designed to capture inter-frame microvascular changes and critical spatial features of facial regions, respectively. To ensure data diversity and reliability, we constructed a novel dataset and conducted extensive experimental validation. Results demonstrate the effectiveness of MT-STFuse, achieving heart rate estimation metrics of MAE (2.03), RMSE (3.12), and MAPE (3.53), and respiration rate estimation metrics of MAE (1.18), RMSE (1.64), and MAPE (7.44), significantly outperforming existing methods. This approach offers a novel and effective solution for developing robust and low-latency non-contact multi-parameter physiological monitoring systems.