This study examines the application of Large Language Models to automatically annotate the phases of the Structural-Demographic Theory from short descriptions of historical decades. This task is useful for understanding social instability, but it is inherently subjective and challenging due to the temporal nature of labels. A single misalignment in phase labeling between annotators can cascade through subsequent time-steps, causing the inter-annotator agreement to decrease exponentially. Our results indicate that models with more than 400 billion parameters achieve very high agreement, while models with fewer than 100 billion parameters are prone to hallucinations. Moreover, the two largest models we tested (GPT4 and Lama3.1-405b) reach inter-annotator agreement comparable to pairs of human annotators, paving the way towards automated annotation. However, the need for very large models could hinder the democratization of automatic historical annotation due to the required computational resources. To mitigate this, we suggest collaborations between universities and companies, in order to share knowledge and computational power.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Large Language Models Rival Human Performance in Historical Labeling

  • Fabio Celli,
  • Valerio Basile

摘要

This study examines the application of Large Language Models to automatically annotate the phases of the Structural-Demographic Theory from short descriptions of historical decades. This task is useful for understanding social instability, but it is inherently subjective and challenging due to the temporal nature of labels. A single misalignment in phase labeling between annotators can cascade through subsequent time-steps, causing the inter-annotator agreement to decrease exponentially. Our results indicate that models with more than 400 billion parameters achieve very high agreement, while models with fewer than 100 billion parameters are prone to hallucinations. Moreover, the two largest models we tested (GPT4 and Lama3.1-405b) reach inter-annotator agreement comparable to pairs of human annotators, paving the way towards automated annotation. However, the need for very large models could hinder the democratization of automatic historical annotation due to the required computational resources. To mitigate this, we suggest collaborations between universities and companies, in order to share knowledge and computational power.