This article investigates how volunteer forecasters, large language models (LLMs) and their combinations can support anticipatory foreign policy analysis in resource-constrained national security contexts. A 9-month quasi-experimental study was conducted with 34 Finnish volunteer forecasters and six LLM-based forecasters, generated by crossing model type (cloud-hosted GPT-4o versus on-premises Gemma 2) with input richness (question only, news summaries, hybrid including human forecasts). In total, 46 geopolitical outcomes were predicted. Forecasts were evaluated both for accuracy, using Brier scores, and for analytical value, through assessments by professional foreign policy analysts. The results show that volunteer forecasters and hybrid configurations achieved essentially the same high level of predictive accuracy, indicating that human input was necessary for producing the most accurate forecasts. At the same time, LLMs consistently generated rationales rated more useful by analysts, with hybrid models again performing best. These findings highlight the complementary strengths of human contextual insight and AI-generated analysis, and demonstrate practical pathways for integrating volunteer forecasters and advanced models into foreign policy workflows, particularly in small states facing rapidly expanding information demands.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Crowds, Machines and Foresight: Combining Volunteer Crowds and Large Language Models to Sharpen Strategic Forecasts

  • Tuomas Husu

摘要

This article investigates how volunteer forecasters, large language models (LLMs) and their combinations can support anticipatory foreign policy analysis in resource-constrained national security contexts. A 9-month quasi-experimental study was conducted with 34 Finnish volunteer forecasters and six LLM-based forecasters, generated by crossing model type (cloud-hosted GPT-4o versus on-premises Gemma 2) with input richness (question only, news summaries, hybrid including human forecasts). In total, 46 geopolitical outcomes were predicted. Forecasts were evaluated both for accuracy, using Brier scores, and for analytical value, through assessments by professional foreign policy analysts. The results show that volunteer forecasters and hybrid configurations achieved essentially the same high level of predictive accuracy, indicating that human input was necessary for producing the most accurate forecasts. At the same time, LLMs consistently generated rationales rated more useful by analysts, with hybrid models again performing best. These findings highlight the complementary strengths of human contextual insight and AI-generated analysis, and demonstrate practical pathways for integrating volunteer forecasters and advanced models into foreign policy workflows, particularly in small states facing rapidly expanding information demands.