Experiments in Modeling Disagreement

This research examines how annotators’ features influence the labeling of sexist content in social media datasets, with a specific focus on the EXIST dataset, which includes direct sexist messages, reports, and descriptions of sexist experiences and stereotypes. By comparing the use of gold labels derived by majority vote with individual annotator labels, we found that incorporating annotator labels into the input tokens enhances model performance in predicting gold labels. Our study further investigates the impact of additionally integrating annotators’ demographic information into BERT models to enhance performance in subjective natural language processing tasks. We find that integrating such demographic data into the input leads to improved model performance.