ATA: Attentive Text Augmentation for Typo-Robust Language Models
摘要
Text data generated by humans in real-world scenarios often contains a substantial amount of noise, including misspellings, typographical errors, and abbreviations. Pre-trained language models (PLMs) often struggle to make accurate predictions when encountering such noise. To handle this issue, we introduce the Attentive Text Augmentation for Typo-Robust Language Models (ATA), leveraging attention weights to compute each word’s contribution to the model prediction. We train PLM with a dataset augmented by ATA and evaluate its performance on datasets containing textual noise. Our experimental results on four text classification datasets demonstrate that the proposed method effectively enhances the model’s robustness against textual noise.