Research on Chinese Word Segmentation Algorithm in the Tobacco Field Based on the BERT-BiLSTM-CRF Model

This study addresses the issue of poor performance in segmenting specialized terms in Chinese texts within the tobacco domain, using traditional segmentation methods. A Chinese word segmentation method in the tobacco field has been proposed with combining bidirectional encoder representations from transformers and bidirectional long short-term memory-conditional random field (BERT-BiLSTM-CRF). Initially, a tobacco domain text annotation corpus is constructed, integrating general and specialized tobacco dictionaries to enhance the BERT model training. This approach takes advantage of BERT’s understanding of language structures and semantic relationships, allowing more effective use of external knowledge. Then, the BiLSTM model serializes the text and further extracts features in context. Finally, the CRF model is utilized for learning word position labels, thus achieving an optimal word segmentation sequence. Experiments on tobacco domain texts and MSRA generic corpora demonstrate F1-scores of 97.18% and 98.46%, respectively, indicating superior segmentation performance and generalization capabilities of this method compared to common approaches.