One of the factors affecting the natural language processing of low-resource languages, such as the Nigerian Igbo language, is the absence of a large monolingual or parallel corpora. This factor is not merely an academic issue; it is an ethical issue. The privileges provided by AI innovations are lacking in communities with low-resource languages. To reduce this gap, we built Igbo corpora with 1,227,620 tokens consisting of different genre, including texts from BBC Igbo, Bible texts (all 66 books), government documents in Igbo, and folklore stories. We carried out preliminary statistical analyses to ensure that the corpora can be used for different NLP analyses. Our analyses included frequency distributions, N-Grams, lexical dispersion plots, and concordance. The analyses with the Igbo corpora yielded good results. We recommend additions to the corpora and further analyses using machine learning and deep learning algorithms.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Building the Low-Resource Igbo Language Corpus

  • Stanley Chinedum Nwoji,
  • Atajan Abdyyev

摘要

One of the factors affecting the natural language processing of low-resource languages, such as the Nigerian Igbo language, is the absence of a large monolingual or parallel corpora. This factor is not merely an academic issue; it is an ethical issue. The privileges provided by AI innovations are lacking in communities with low-resource languages. To reduce this gap, we built Igbo corpora with 1,227,620 tokens consisting of different genre, including texts from BBC Igbo, Bible texts (all 66 books), government documents in Igbo, and folklore stories. We carried out preliminary statistical analyses to ensure that the corpora can be used for different NLP analyses. Our analyses included frequency distributions, N-Grams, lexical dispersion plots, and concordance. The analyses with the Igbo corpora yielded good results. We recommend additions to the corpora and further analyses using machine learning and deep learning algorithms.