Existing open-source text language identification tools are not ideal for recognizing short texts in Chinese, Japanese, and Korean, and some of them cannot distinguish between Simplified and Traditional Chinese. To improve identification efficiency, this paper proposes a two-stage language identification algorithm for Chinese, Japanese, and Korean. The first stage is based on a character script identification algorithm, according to the Unicode encoding range of Chinese, Japanese, and Korean, and determines the language. If the identification result is Chinese, the classification algorithm developed using the bidirectional long short-term memory (BiLSTM)-Attention model is used in the second stage to determine whether the language is Simplified or Traditional Chinese. The experimental results show that the proposed method improves the identification efficiency of Chinese, Japanese, and Korean short texts compared to existing open-source language identification tools.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Research on Two-Stage Text Language Identification Algorithm for Chinese, Japanese, and Korean

  • Mamtimin Qasim,
  • Wushour Silamu,
  • Minghui Qiu

摘要

Existing open-source text language identification tools are not ideal for recognizing short texts in Chinese, Japanese, and Korean, and some of them cannot distinguish between Simplified and Traditional Chinese. To improve identification efficiency, this paper proposes a two-stage language identification algorithm for Chinese, Japanese, and Korean. The first stage is based on a character script identification algorithm, according to the Unicode encoding range of Chinese, Japanese, and Korean, and determines the language. If the identification result is Chinese, the classification algorithm developed using the bidirectional long short-term memory (BiLSTM)-Attention model is used in the second stage to determine whether the language is Simplified or Traditional Chinese. The experimental results show that the proposed method improves the identification efficiency of Chinese, Japanese, and Korean short texts compared to existing open-source language identification tools.