In recent years, photoplethysmography (PPG) signals have garnered significant attention for blood pressure monitoring. However, notable prediction errors persist, particularly in the prediction of systolic blood pressure. Transformer-based methods for time series analysis have emerged, with a focus on tokenizing time series data. However, few researchers have integrated information from both the time and frequency domains, despite PPG signals being time-series data. We propose Fre-CrossFormer, a neural architecture that utilizes frequency-domain information to supervise feature learning in continuous non-invasive blood pressure monitoring. Firstly, the raw data are encoded into tokens through the encoding module, where our multi-scale convolutional layers effectively address the multi-periodic characteristics of PPG signals. Secondly, we extract frequency-domain information via the Fourier transform and employ cross-attention mechanisms to guide the model's interaction with spectral features. Finally, frequency domain information is systematically integrated into the transformer architecture to enhance feature representation. Experiments carried out on the MIMIC-II dataset demonstrate the efficacy of Fre-CrossFormer, achieving a mean absolute error (MAE) of 2.54 mmHg and a standard deviation (SD) of 4.48 mmHg for diastolic BP, while systolic BP has a MAE of 2.65 mmHg and an SD of 4.71 mmHg. The test results meet the highest standards set by the Association for the Advancement of Medical Instrumentation (AAMI) and the British Hypertension Society (BHS). Furthermore, our model performs exceptionally well on time series classification tasks, achieving accuracy rates of 92.19% and 98.25% in the Human Activity Recognition (HAR), Sleep-EDF, and Epilepsy datasets, respectively. These results demonstrate the reliability and robustness of the Fre-CrossFormer, paving the way for future research and practical applications in the field.