Staged Long Text Generation with Progressive Task-Oriented Prompts

Generating coherent and consistent long text remains a challenge for artificial intelligence. The state-of-the-art paradigm partitions the whole generating process into successive stages, however, the content plan applied in each stage may be error-prone and fine tuning large-scale language models, one for each stage, is resource-consuming. In this paper, we follow the above paradigm and devise three stages: keyphrase decompression, transition paraphrase, and text generation. We leverage task-oriented prompts to direct the producing of text in each stage which improves the quality of the generated text. Further, we propose a new content plan representation with elastic mask tokens to reduce model bias and irregular words. Moreover, we introduce length control and commonsense knowledge prompts to increase the adaptability of the proposed model. Extensive experiments conducted on two challenging tasks demonstrated that our model outperforms strong baselines significantly, and it is able to generate longer high quality texts with fewer parameters.