Research on Two-Stage Text Language Identification Algorithm for Chinese, Japanese, and Korean
摘要
Existing open-source text language identification tools are not ideal for recognizing short texts in Chinese, Japanese, and Korean, and some of them cannot distinguish between Simplified and Traditional Chinese. To improve identification efficiency, this paper proposes a two-stage language identification algorithm for Chinese, Japanese, and Korean. The first stage is based on a character script identification algorithm, according to the Unicode encoding range of Chinese, Japanese, and Korean, and determines the language. If the identification result is Chinese, the classification algorithm developed using the bidirectional long short-term memory (BiLSTM)-Attention model is used in the second stage to determine whether the language is Simplified or Traditional Chinese. The experimental results show that the proposed method improves the identification efficiency of Chinese, Japanese, and Korean short texts compared to existing open-source language identification tools.