The stock market plays a crucial role in the development of the country and society, whose development status reflects the overall trend of the national economy. It is eager to predict the future stock price correctly for investors in the market. However, the stock price by itself has noisy, volatile and nonlinear characteristics, making the effective prediction challenging. In this paper, we propose a transformer-based model with the patch and gate attention mechanisms, named Patch-GAU, to predict the stock price effectively. Patch-GAU uses the patch mechanism to reduce the impact of abnormal data points on the gate attention calculation by encoding time series data in blocks. Besides, Patch-GAU introduces gate attention units to simplify the model structure while retaining the attention mechanism and reducing the computational overhead of the self-attention mechanism in the base transformer model. Moreover, Patch-GAU applies the idea of probabilistic sparsity for the attention calculation to further reduce the computational complexity of the model. The experimental results compared with benchmark models such as Transformer and Informer, show that our model outperforms these benchmark models in stock price prediction.