N-Gram Language Model for Uzbek Texts
摘要
This article presents language models, specifically the statistical method known as the N-gram model. Language models are mathematical or computational models that probabilistically represent sequences of words in natural languages (for example, Uzbek, English). They are used to predict the next word based on a given context, generate text, assess word probabilities or analyze language-specific statistical features. The following smoothing methods used in the N-gram model are discussed: Laplace smoothing, Good-Turing discounting, Katz back-off, Interpolation, and Kneser-Ney smoothing. The article presents the stages of building an n-gram language model for Uzbek texts: corpus preparation, N-gram formation, smoothing, language model format preparation, and prediction mechanisms. The corpus frequencies, i.e., the distribution of N-gram counts, are presented both in logarithmic scale and in linear graph form. As a result, when applying Kneser–Ney smoothing for the Uzbek language, the test perplexity of the 3-gram model was 121, for the 5-gram model it was 114, for the 7-gram model it was 112, and for the 9-gram model it was 108.