Chinese Lexical Analysis via Era-Aware Large Language Models
摘要
The core tasks of Chinese lexical analysis are word segmentation and part-of-speech tagging. Existing tools are usually trained on corpora from a single period, resulting in high error rates when applied to texts from other historical stages. This study introduces an era-aware training method that explicitly incorporates temporal features. Large language models are fine-tuned on corpora representing four historical periods of Chinese: Ancient, Middle, Early Modern, and Modern. Experimental results show that the inclusion of temporal information substantially improves lexical analysis performance in all four eras. Among the evaluated models, Qwen-2-7B achieved the best overall results, with F1 scores of 97.58% (word segmentation) and 95.38% (POS) in Ancient Chinese, and 99.34% (word segmentation) in Modern Chinese. These findings demonstrate that temporal features are essential for enhancing both the accuracy and robustness of lexical analysis across different historical stages of Chinese.