Under-Sampling Strategies for Better Transformer-Based Classifications Models

This paper presents findings from the Check-That! Lab Task 1B-English submission at CLEF 2023. The research developed a method for evaluating the check-worthiness of short English texts. The first iteration focused on identifying optimal model architectures and adaptation techniques, while the second iteration involved curating the dataset for improved results. The study included fine-tuning several GPT and BERT models, applying zero-shot, few-shot, and Chain-of-Thought prompting strategies, and utilizing dataset sampling techniques informed by quality and training dynamics metrics. Team achieved first place in the competition by fine-tuning the OpenAI GPT-3 curie model. Findings suggest that fine-tuned BERT models can perform comparably to GPT models, but dataset curation was pivotal in obtaining superior results across various model architectures.