SFMD: A Semi-supervised Framework for Pre-trained Language Models Fine-Tuning with Noisy Samples

Fine-tuning language models with noisy samples carries the risk of causing pre-trained models to overfit to incorrect information, leading to a decline in performance for specific tasks. Securing datasets tailored for specific tasks without any noise is difficult and infrequent, which poses challenges to practical model training in real-world settings. To address the challenge of noisy samples, we propose a Semi-supervised Framework incorporating a sentence-level MixUp technique for the supervised module and word-level Data Augmentation for the self-supervised module (SFMD) for text classification tasks. The supervised learning module with the MixUp technique is designed to prevent memorization of noise and the self-supervised learning module is designed to learn the intrinsic similarity of samples and mitigate the influence of noise. The traditional label-dependent supervised module and constraints of the feature-dependent self-supervised module are applied to language models, and the network is regularized to maximize the consistency of the two constraints. In experiments conducted with corrupted data sourced from benchmark datasets, our method surpasses baseline approaches, showcasing significant performance improvements on noisy data. Particularly noteworthy is the progressive enhancement in our models’ performance as the level of noise increases.