The Transformer model has excelled in multivariate time series forecasting by using channel-wise self-attention. However, it has limitations, such as a lack of temporal constraints when extracting temporal features and ineffective use of cumulative historical data. To overcome these issues, we introduce the Structured Channel-wise Transformer with Cumulative Historical state (SCFormer). SCFormer adds temporal constraints to all linear transformations, including the query, key, and value matrices, as well as the fully connected layers in the Transformer. It also uses High-order Polynomial Projection Operators (HiPPO) to effectively incorporate cumulative historical data, enabling the model to utilize information beyond the look-back window during predictions. Extensive experiments on various real-world datasets show that SCFormer significantly outperforms leading baselines, demonstrating its effectiveness in improving time series forecasting. The code is publicly available at https://github.com/ShiweiGuo1995/SCFormer