Efficient Fine-Tuning for Low-Resource Tibetan Pre-trained Language Models

For low-resource languages like Tibetan, the availability of pre-trained language models (PLMs) is severely limited both in quantity and performance. Therefore, it is crucial to explore the optimization of these limited PLMs. In this paper, leveraging the downstream tasks of Tibetan news title classification task provided by the Tibetan News Classification Corpus (TNCC), we conducted optimization experiments on multiple Tibetan PLMs using prompt learning and LoRA (Low-Rank Adaptation) efficient fine-tuning techniques. We mainly conducted two types of experimental investigations: full-shot and few-shot for prompt learning. The full-shot experiment demonstrate that prompt learning improves the classification performance of PLMs, while LoRA achieves a significant reduction in the number of trainable parameters with very minor performance degradation for PLMs. Notably, from the few-shot experiments, we observe that prompt learning significantly enhances the classification performance of PLMs in low-resource scenarios.