Automated Text Corpus Development Using Web Crawling for Low-Resource Mizo Language: A Foundation for NLP Research
摘要
Mizo, the official language of Mizoram State, India is a low-resource language with limited resources. The scarcity of a large monolingual corpus poses a significant challenge for Mizo natural language processing (NLP) tasks. This paper presents the development of the largest Mizo text corpus to date using web crawling techniques. This extensive monolingual corpus is important for many NLP tasks such as word embedding, text summarization, entity recognition, tagging of parts of speech, translation, and sentiment analysis. It also helps preserve valuable Mizo writings and content that might otherwise be lost owing to the inactivity of the blog, deletion over time, or expired hosting services. The creation of this largest Mizo text corpus consisting of 228,367 text documents establishes a significant foundation for computational linguistic research in Mizo, enhancing its digital representation and providing a vital resource for future research.