Contemplating Dialects When Building a Guarani Corpus for NLP

In contemporary times, grammars and dictionaries for minority languages of South America have seen increased publication, serving as invaluable resources for language scholars. It is important to note that the term “minority” here does not pertain to the number of speakers but rather the under-represented status of these languages. However, Guarani, despite being one of the most widely spoken and researched South American languages, has received relatively little attention from a technological viewpoint. This paper delves into the unique challenges that computational linguists encounter when dealing with Guarani. These challenges include the scarcity of bilingual texts necessary to construct a linguistic corpus, its coexistence with Spanish, leading to the emergence of new linguistic variations and code-switching, and the inconsistent and varied spelling practices within the Guarani language. The paper elucidates how these issues were navigated during the development of a parallel corpus of Guarani-Spanish texts, meticulously aligned at the sentence level. In doing so, the paper offers insights into the intricate linguistic and natural language processing (NLP) considerations involved when working with a minority South American language. This analysis is underpinned by a thorough examination of the Jojajovai corpus (Chiruzzo, Luis, Santiago Góngora, Aldo Alvarez, Gustavo Giménez-Lugo, Marvin Agüero-Torales, and Yliana Rodríguez (2022), Jojajovai: A parallel guarani-spanish corpus for mt benchmarking, Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 2098–2107.), a comprehensive collection comprising approximately 30,000 sentence pairs. This corpus draws from diverse sources and is thoughtfully partitioned into training, development, and test subsets. In essence, this work encapsulates the manifold challenges encountered in working with a minority South American language and underscores the linguistic and NLP complexities that such endeavors entail.