Large Language Models Rival Human Performance in Historical Labeling
摘要
This study examines the application of Large Language Models to automatically annotate the phases of the Structural-Demographic Theory from short descriptions of historical decades. This task is useful for understanding social instability, but it is inherently subjective and challenging due to the temporal nature of labels. A single misalignment in phase labeling between annotators can cascade through subsequent time-steps, causing the inter-annotator agreement to decrease exponentially. Our results indicate that models with more than 400 billion parameters achieve very high agreement, while models with fewer than 100 billion parameters are prone to hallucinations. Moreover, the two largest models we tested (GPT4 and Lama3.1-405b) reach inter-annotator agreement comparable to pairs of human annotators, paving the way towards automated annotation. However, the need for very large models could hinder the democratization of automatic historical annotation due to the required computational resources. To mitigate this, we suggest collaborations between universities and companies, in order to share knowledge and computational power.