Framing and BERTology: A Data-Centric Approach to Integration of Linguistic Features into Transformer-Based Pre-trained Language Models

Our study tackles the concept of framing in political news media, its linguistic realization, and the integration of the latter into BERT. To this end, we utilized a human-labeled dataset of news articles annotated with the issue and game frame along with deep linguistic features capturing this frame. We fine-tuned BERT on a binary classification task adding the features to the input. We demonstrate which specific linguistic categories seem to be comprehended by BERT and should be paid more attention to in the further development of BERT-like language models.