Fusion of Root and Affix Information with Pre-trained Language Models for Text Classification

In the English language, words sharing the same root or affix often demonstrate a high level of semantic relatedness, and their positions in the semantic space are relatively proximate, primarily due to their shared origin. However, existing pre-trained language models based on the Transformer framework typically employ a strategy of segmenting long words into subwords, which does not comprehensively capture all the root and affix information. This limitation, to some extent, constrains the enhancement of text classification methods. In light of this, we introduce a novel text classification method, termed Fusion of Root and Affix Information with Pre-trained Language Models for Text Classification (FusionRA). FusionRA aims to acquire more comprehensive semantic information by amalgamating the information of roots and affixes with the word-level features extracted from pre-trained models. Specifically, FusionRA utilizes two distinct Transformer Encoders: one for extracting information on roots and affixes, and the other for extracting semantic features of samples. By leveraging the extracted root and affix information to conduct secondary training on the pre-trained model, the model learns the characteristics of roots and affixes. Subsequently, the semantic information of these roots and affixes is fused with the semantic information of the samples. Ultimately, through joint training, FusionRA achieves precise text classification. To validate the efficacy of FusionRA, we conducted experimental studies on six benchmark text classification datasets. The experimental results demonstrate a significant enhancement in the classification accuracy of FusionRA, affirming the superiority of this method in text classification tasks.