STATS-BERT: An Enhanced BERT Model with Statistical Masking for Improved Transfer Performance

With the advent of transfer learning methods in NLP, many state-of-the-art architectures involve pretrained models, which are put to use for many heterogeneous tasks through the process of finetuning. These finetuned models give better results when compared to task-specific models because of its extensive pretraining. BERT is one such powerful pretrained model that has marked a significant improvement over a plethora of downstream tasks which includes text classification, Question-Answering, etc. In light of these transfer learning approaches, we introduce an improvised version of BERT called STATS-BERT, based on a statistical masking objective. The STATS-BERT model incorporates statistical properties such as TF-IDF and Entropy information to develop a more resilient model in terms of capturing semantic information. The proposed BERT variant, STATS-BERT is found to outperform the analogous baseline model in 4 out of 6 tasks empirically on similar experimental grounds. In particular, STATS-BERT shows 0.6% increase when compared to the baseline model on question classification dataset (TREC).