In the field of Natural Language Processing (NLP), an excellent evaluation framework is crucial to the development of target tasks or fields. The emergence of Large Language Models (LLMs) requires some effective evaluation frameworks to protect their development. Traditional NLP tasks can be used to test the ability to understand and use language, but an evaluation framework that can test the accuracy and conciseness of language expression is still needed, that is, a dataset that can evaluate the ability to use long-tail senses. In addition, the Word Sense Disambiguation (WSD) task has benefited from some excellent evaluation frameworks and has been steadily developed. As models effectively identify high-frequency senses, the research focus of the WSD task has shifted to the identification of low-frequency senses, that is, long-tail senses. Based on the original evaluation framework of WSD, this paper constructs an evaluation framework that distinguishes high- and low-frequency senses, and the evaluation framework can be used to evaluate the vocabulary-level language understanding and expression ability of long-tail senses of LLMs, as well as the recognition ability of long-tail senses of WSD models.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

An Evaluation Framework for Long-Tail Senses in Large Language Models and Word Sense Disambiguation

  • Junwei Zhang,
  • Tianheng Wang,
  • Tao Huang,
  • Yupeng Zhang,
  • Pengju Yan,
  • Xiaolin Li

摘要

In the field of Natural Language Processing (NLP), an excellent evaluation framework is crucial to the development of target tasks or fields. The emergence of Large Language Models (LLMs) requires some effective evaluation frameworks to protect their development. Traditional NLP tasks can be used to test the ability to understand and use language, but an evaluation framework that can test the accuracy and conciseness of language expression is still needed, that is, a dataset that can evaluate the ability to use long-tail senses. In addition, the Word Sense Disambiguation (WSD) task has benefited from some excellent evaluation frameworks and has been steadily developed. As models effectively identify high-frequency senses, the research focus of the WSD task has shifted to the identification of low-frequency senses, that is, long-tail senses. Based on the original evaluation framework of WSD, this paper constructs an evaluation framework that distinguishes high- and low-frequency senses, and the evaluation framework can be used to evaluate the vocabulary-level language understanding and expression ability of long-tail senses of LLMs, as well as the recognition ability of long-tail senses of WSD models.