Generative large language models (LLMs), such as GPT-4, have demonstrated remarkable performance across a wide range of NLP tasks. The increased number of LLMs’ parameters enhances their generalization capabilities, but it also results in a higher computational burden and slower inference speed. Addressing these concern, through continued pre-training, we have enhanced the BERT model (based on MLM and NSP pre-training tasks) to achieve performance comparable to that of the large Qwen-1.5 series models in zero-shot and few-shot text classification tasks, demonstrating that non-generative pre-trained models like BERT are still relevant. Additionally, by using LLMs to expand topic words related to downstream tasks and employing word density methods to retrieve corresponding subsets from large corpora, we have obtained better pre-training results.