ESSformer: Transformers with ESS Attention for Long-Term Series Forecasting

The transformer’s global modeling capability makes it crucial for long-time series forecasting (LTSF). We propose an effective Transformer-based model for LTSF: ESSformer. It is built upon two essential components: (i) We adopt the Channel-Patch Independence architecture, where channels share the same model weights and have independent embeddings to avoid the impact of distribution shifts between channels. Patches are used to extract local semantic information from the time series. A self-supervised representation learning strategy is introduced to enhance feature representation capabilities. Initially, We randomly remove a portion of the input sequence, then train the ESSformer model to restore the missing content. By reconstructing using the unmasked patches, this model demonstrates better overall robustness to distribution shifts and can be traced back to longer histories. (ii) We design Extended Sparse Smooth (ESS) Attention, which employs downsampling to halve the inputs of the cascade layer, highlights the main attention, and can efficiently deal with ultra-long input sequences. It considers different heads at the same time, blending low and high-rise features to obtain multiple sets of attention results, which are then spliced and linearly projected to obtain the final output. This method can effectively capture the local temporal dependencies between various patches and channels, enabling a comprehensive data representation. In experiments, ESSformer demonstrates significant superiority over seven mainstream methods, offering a novel solution for LTSF.