EBFP: Example-Based Further Pre-training

By conducting effective further pre-training before fine-tuning the Pre-trained Language Models (PLMs), the performance of downstream tasks can be further improved. In many few-shot tasks, the effect of processing downstream tasks through prompt learning has already surpassed traditional fine-tuning. Since prompt learning requires packaging data with templates, the previously used further pre-training method suitable for fine-tuning is difficult to learn the relationship between templates and data. Large language models can better stimulate their potential by adding examples to prompts. In fact, even for small PLMs (with model parameters less than 1B), adding examples to the input can also improve their performance in tasks. Therefore, this study proposes a new further pre-training method called Example-Based Further Pretraining (EBFP). To enable small PLMs to better understand the use of examples, this study introduces prompt information and example information into the PLMs to improve the performance of prompt learning in downstream tasks. In order to test the effectiveness of EBFP, this study applied EBFP to different prompt methods. Experimental results show that compared with traditional further pre-training method, the EBFP method can more effectively enhance the effect of prompt learning in downstream tasks.