PYGC: A PinYin Language Model Guided Correction Model for Chinese Spell Checking

Chinese Spell Checking (CSC) is an NLP task that detects and corrects erroneous characters in Chinese texts. Since people often use pinyin (pronunciation of Chinese characters) input methods or speech recognition to type text, most of these errors are misuse of phonetically or semantically similar characters. Previous attempts fuse pinyin information into the embedding layer of pre-trained language models. However, although they can learn from phonetic knowledge, they can not make good use of this knowledge for error correction. In this paper, we propose a PinYin language model Guided Correction model (PYGC), which regards the Chinese pinyin sequence as an independent language model. Our model builds on two parallel transformer encoders to capture pinyin and semantic features respectively, with a late fusion module to fuse these two hidden representations to generate the final prediction. Besides, we perform an additional pronunciation prediction task on pinyin embeddings to ensure the reliability of the pinyin language model. Experiments on the widely used SIGHAN benchmark and a newly released CSCD-IME dataset with mainly pinyin-related errors show that our method outperforms current state-of-the-art approaches by a remarkable margin. Furthermore, isolation tests demonstrate that our model has the best generalization ability on unseen spelling errors. (Code available in https://github.com/Imposingapple/PYGC ).