Measuring Bias in Generated Text Using Language Models—GPT-2 and BERT

In Natural Language Processing (NLP), a language model is a probabilistic statistical model that estimates the likelihood that a particular sequence of words will appear in a sentence based on the words that came before it. In our experiment, text prompts for evaluating bias were generated using the BERT and GPT-2 language models. Sentiment, toxicity, and gender polarity are the bias measures that we have added in order to measure biases from numerous perspectives. During fine-tuning BERT model, we have achieved 91.48% accuracy on multilabel toxic comment classification. Later, this fine-tuned pretrained model is used for generating text using BOLD dataset prompts. Our work shows a greater percentage of the texts produced by GPT-2 than those produced by BERT which were labeled as toxic. Similar to how it did in the religious ideology sector, BERT's communism prompt resulted in a toxic text. Compared to BERT, GPT-2 produced writings that were more polarized in terms of sentiment, toxicity, and regard.