Named Entity Recognition on Ancient Languages Using Large Language Models: An Exploratory Study
摘要
The historiography of the ancient world largely focuses on prominent individuals, states, and empires as a tradition originating from 5th century BCE Greek historians like Herodotus and Thucydides. Despite longstanding, abundant cataloging of significant figures and places, there is a notable gap in the comprehensive cataloging of the numerous groups that have existed throughout recorded history. The Herodotos Project, an initiative aimed at filling this gap by employing Named Entity Recognition (NER) systems to identify groups in Classical texts, enables us to study the potential of leveraging ChatGPT for this task. We examine its performance in tagging ancient Latin texts. We find that ChatGPT’s performance increases with decreased text length and peaks at a value higher than that of other, specialized algorithms. Results for known texts achieve an F-1 score of 0.99 for untagged words, 0.81 for geographical entities, 0.84 for group entities, and 0.87 for persons, which rivals but falls short of the 0.9 score from previous work done with a supervised model by the Herodotos project. Applying this to novel Latin text yields an F-1 score of 0.97 for untagged words, 0.52 for geographical entities, 0.62 for group entities, and 0.60 for persons. This indicates ChatGPT’s effectiveness in tagging known texts while highlighting challenges with newer, less familiar texts. This small study also suggests avenues for further research into model performance across diverse languages and text corpora.