Controllable Story Generation Based on Perplexity Minimization

Large-scale pre-trained language models have demonstrated impressive results in producing human-like texts. However, controlling the text generation process remains a challenge for researchers. Controllable text generation consists of generating sentences that satisfy desired constraints (e.g., sentiment, topic, or keywords). Recent studies that control the decoding stage of a language model have proved the high efficiency of this approach for control of generated texts. This approach, in contrast to the fine-tuning of pre-trained language models, requires much less computing resources. In this work, we propose and investigate a method that controls the process of language generation using perplexity minimization. The method is designed to create stories from a sequence of guide phrases that form a storyline and is based on the search for sequences of tokens that reduce text perplexity when generation is directed towards the guide phrase. First, we generate several arbitrary small sequences of tokens from the language model vocabulary. Then we choose the most probable subsequence - the one, the probability of following the guide phrase after which is the biggest. The proposed method induces the model to shift the content of the generated text to the guide phrase. Experiments on the Russian-language corpus of fairy tales with storylines have shown the high efficiency of the proposed method for creating stories corresponding to the user-specified storyline.