FEDAF: frequency enhanced decomposed attention free transformer for long time series forecasting

Long time series forecasting (LTSF), which involves modeling relationships within long time series to predict future values, has extensive applications in domains such as weather forecasting, financial analysis, and traffic prediction. Recently, numerous transformer-based models have been developed to address the challenges in LTSF. These models employ methods such as sparse attention to alleviate the inefficiencies associated with the attention mechanism and utilize decomposition architecture to enhance the predictability of the series. However, these complexity reduction methods necessitate additional calculations, and the series decomposition architecture overlooks the random components. To overcome these limitations, this paper proposes the Frequency Enhanced Decomposed Attention Free Transformer (FEDAF). FEDAF introduces two variants of the Frequency Enhanced Attention Free Mechanism (FEAFM), namely FEAFM-s and FEAFM-c, which seamlessly replace self-attention and cross-attention. Both variants perform calculations in the frequency domain without incurring additional costs, with the time and space complexity of FEAFM-s being \(\mathcal {O}(L{\text {log}}L)\) $\mathcal{O}(L\text{log}L)$ . Additionally, FEDAF incorporates a time series decomposition architecture that considers random components. Unlike other models that solely decompose the series into trend and seasonal components, FEDAF also eliminates random terms by applying Fourier denoising. Our study quantifies data drift and validates that the proposed decomposition structure can mitigate the adverse effects caused by data shift. Overall, FEDAF demonstrates superior forecasting performance compared to state-of-the-art models across various domains, achieving a remarkable improvement of 19.49% for Traffic in particular. Furthermore, an efficiency analysis reveals that FEAFM enhances space efficiency by 12.8% compared to the vanilla attention mechanism and improves time efficiency by 43.63% compared to other attention mechanism variants.