Evaluating Agentic AI Through a Geopolitical Simulation Sandbox
摘要
Intelligent agents are the technology of the moment, with their ability to interact in their environment according to the objectives set for them. Their immense potential makes them a disruptive technology, and they will undoubtedly have a global impact on the job market. This technology will bring its share of benefits, but also its share of challenges. With this in mind, it is crucial to understand, evaluate, and circumscribe this technology to optimise its positive effects while also mitigating its adverse effects. This is no easy task, since agents, especially generative agents, are not deterministic, and their quantitative evaluation poses a challenge. This paper presents a geopolitical simulator used as a testbed for the development, evaluation, and circumscription of Large Language Model (LLM) based intelligent agents. Using a fictitious world map of 20 countries enables agents and their hierarchies to interact, communicate, negotiate, and collaborate to achieve their goals. Agents’ behaviours and decisions are observed, evaluated, and quantified using methods that implement metrics to describe their level of ethics, collaboration, negotiation, creativity, and so on. The simulation is carried out using a hierarchy of classes, instantiated as objects, whose different variables have a causal and cascading impact on one another. Eventually, the various agents, based on popular LLMs (Chat GPT, Claude, Mistral, and others), will be compared and classified according to each of the metrics, providing the community with more in-depth knowledge of each of the major models.