The historiography of the ancient world largely focuses on prominent individuals, states, and empires as a tradition originating from 5th century BCE Greek historians like Herodotus and Thucydides. Despite longstanding, abundant cataloging of significant figures and places, there is a notable gap in the comprehensive cataloging of the numerous groups that have existed throughout recorded history. The Herodotos Project, an initiative aimed at filling this gap by employing Named Entity Recognition (NER) systems to identify groups in Classical texts, enables us to study the potential of leveraging ChatGPT for this task. We examine its performance in tagging ancient Latin texts. We find that ChatGPT’s performance increases with decreased text length and peaks at a value higher than that of other, specialized algorithms. Results for known texts achieve an F-1 score of 0.99 for untagged words, 0.81 for geographical entities, 0.84 for group entities, and 0.87 for persons, which rivals but falls short of the 0.9 score from previous work done with a supervised model by the Herodotos project. Applying this to novel Latin text yields an F-1 score of 0.97 for untagged words, 0.52 for geographical entities, 0.62 for group entities, and 0.60 for persons. This indicates ChatGPT’s effectiveness in tagging known texts while highlighting challenges with newer, less familiar texts. This small study also suggests avenues for further research into model performance across diverse languages and text corpora.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Named Entity Recognition on Ancient Languages Using Large Language Models: An Exploratory Study

  • Charles Woodrum,
  • Brian Joseph,
  • Antoine Haaker,
  • Christopher Brown

摘要

The historiography of the ancient world largely focuses on prominent individuals, states, and empires as a tradition originating from 5th century BCE Greek historians like Herodotus and Thucydides. Despite longstanding, abundant cataloging of significant figures and places, there is a notable gap in the comprehensive cataloging of the numerous groups that have existed throughout recorded history. The Herodotos Project, an initiative aimed at filling this gap by employing Named Entity Recognition (NER) systems to identify groups in Classical texts, enables us to study the potential of leveraging ChatGPT for this task. We examine its performance in tagging ancient Latin texts. We find that ChatGPT’s performance increases with decreased text length and peaks at a value higher than that of other, specialized algorithms. Results for known texts achieve an F-1 score of 0.99 for untagged words, 0.81 for geographical entities, 0.84 for group entities, and 0.87 for persons, which rivals but falls short of the 0.9 score from previous work done with a supervised model by the Herodotos project. Applying this to novel Latin text yields an F-1 score of 0.97 for untagged words, 0.52 for geographical entities, 0.62 for group entities, and 0.60 for persons. This indicates ChatGPT’s effectiveness in tagging known texts while highlighting challenges with newer, less familiar texts. This small study also suggests avenues for further research into model performance across diverse languages and text corpora.