Recently, Transformer-based generative models have made remarkable advancements in various domains. However, the generation quality and inference stability of existing approaches are unsatisfactory due to the local overfitting problem. To this end, we propose a novel Span Generation and Denoise Generation strategy, SGDG, to alleviate this problem. Span Generation enhances the model’s ability to globally fit the target text by predicting a continuous segment (span) simultaneously using the span attention mechanism. Additionally, we incorporate Denoise Generation, which randomly replaces the token from the most recent step with a prediction noise, to prevent the model from excessively relying on local historical information. Our extensive experiments on three tasks (dialogue generation, summarization, and question generation) demonstrate improved generation quality of Transformer Seq2Seq models with the proposed SGDG over existing strategies.