Time series analysis has various applications in different fields, such as traffic management, weather forecasting, and crime prevention. Currently, deep learning-based models, particularly those employing self-attention mechanisms like transformer methods, are achieving remarkable results for time series prediction. However, the current models are not sophisticated enough to accurately model time series due to the non-stationarity of time series data and challenges in merging information. To address these issues, this study proposes a novel approach: the Frequency Fusion Network (FFN). FFN learns the mapping from the frequency domain to the time domain, utilizing frequency convolution to merge information from different patterned time series while preserving their statistical characteristics. Experimental results demonstrate that FFN consistently outperforms existing methods across all datasets, and offers a fresh perspective and effective methodology for improving time series forecasting accuracy, especially in handling non-stationary data.