GenerCTC: a general two-stage contrastive training framework for text classification

Contrastive learning methods have been widely applied to text classification. However, the training objective of traditional contrastive learning is different from that of classification because introducing contrastive learning may weaken clustering structures in the latent space. Simultaneously, employing cross-entropy loss in classification training results in learning only discrete features and limits the model’s learning ability. This paper proposes a general two-stage contrastive training framework integrating a novel contrastive learning and classification training method. Meanwhile, we introduce a constraint term for the training objective of contrastive learning to enable the model to learn appropriate levels of instance-level knowledge, which facilitates subsequent classification tasks. Furthermore, In the classification training stage, we present a novel classification loss function that concentrates on correct class predictions and strengthens the associations between incorrect and correct classes, thus enabling the model to learn continuous features and better explore the implicit relationships between classes. In addition, we also optimize the weight initialization of the classifier to improve classification performance. Our experiments on various text classification benchmarks and some challenging few-shot classification tasks demonstrate the effectiveness of our proposed method.