Paraphrasers and Classifiers: Controllable Text Generation for Text Style Transfer

Text style transfer (TST) is an NLP task with a long history and a broad range of applications. Recently, it has seen success with the use of large pretrained language models (LMs). However, the size of contemporary LMs often makes fine-tuning for downstream tasks infeasible. For this reason, methods of controllable text generation (CTG) which do not aim at fine-tuning the original LM have received attention for solving TST tasks. In this work, we contribute to this line of research and adapt an existing CTG method, CAIF, for TST. The original CAIF is based on reweighting the logits of the generative LM according to a free-form style attribute classifier. To allow its use for TST, we replace the standard LM with a model capable of paraphrasing, making corresponding changes. We refer to the resulting unsupervised method as ParaCAIF. We illustrate its applicability by experimenting with detoxification, a relatively new yet practical TST subtask. We work with detoxification in two languages: Russian and English. For both languages, ParaCAIF significantly reduces the toxicity of the generated paraphrases as compared to plain paraphrasers. To the best of our knowledge, it is the first work that adapts a CTG method for Russian detoxification. For English, ParaCAIF outperforms an analogous adapted CTG method, ParaGeDi, in terms of style transfer accuracy. Although the overall performance of ParaCAIF remains lower than that of supervised approaches, it has a broader range of application, as it does not require parallel data which are not readily available for many TST tasks.