The Google Book Ngram corpus is captivating due to its incredible size and availability. It has been widely used in studies of culture, social psychology, language evolution and others. However, as some researchers state, it suffers a number of limitations. Apparently, the most serious limitation of the corpus is its genre imbalance and lack of information on the genre composition of the books included in it. In this paper, we developed an algorithm for estimating the genre composition of the Google Books Ngram corpus. To estimate the percentage of texts of different genres, we used data on relative frequencies of a large range of words. Both linear models and multilayer feedforward neural networks were tested as predictors. To train the predictors, we used random subsamples of texts from the COHA corpus which are marked up by genres. We obtained estimates by using linear predictors and neural network predictors which showed different effectiveness. To assess the achieved accuracy, a cross-validation was performed. The analysis showed that the standard deviation of the neural network estimates obtained from annual data is no worse than 2–2.2%. The constructed estimates of the genre composition of Google Books Ngram also response to major historical events. It should also be noted that the genre composition has changed significantly since 2008. The obtained results provide a vision of the Google Book Ngram corpus genre composition and offer a possible framework for improvements to future works based on the Google Book Ngram data.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Estimation of the Genre Composition of the English Subcorpus of the Google Books Ngram

  • Vladimir Bochkarev,
  • Andrey A. Achkeev,
  • Anna Shevlyakova

摘要

The Google Book Ngram corpus is captivating due to its incredible size and availability. It has been widely used in studies of culture, social psychology, language evolution and others. However, as some researchers state, it suffers a number of limitations. Apparently, the most serious limitation of the corpus is its genre imbalance and lack of information on the genre composition of the books included in it. In this paper, we developed an algorithm for estimating the genre composition of the Google Books Ngram corpus. To estimate the percentage of texts of different genres, we used data on relative frequencies of a large range of words. Both linear models and multilayer feedforward neural networks were tested as predictors. To train the predictors, we used random subsamples of texts from the COHA corpus which are marked up by genres. We obtained estimates by using linear predictors and neural network predictors which showed different effectiveness. To assess the achieved accuracy, a cross-validation was performed. The analysis showed that the standard deviation of the neural network estimates obtained from annual data is no worse than 2–2.2%. The constructed estimates of the genre composition of Google Books Ngram also response to major historical events. It should also be noted that the genre composition has changed significantly since 2008. The obtained results provide a vision of the Google Book Ngram corpus genre composition and offer a possible framework for improvements to future works based on the Google Book Ngram data.