Intelligent agents are the technology of the moment, with their ability to interact in their environment according to the objectives set for them. Their immense potential makes them a disruptive technology, and they will undoubtedly have a global impact on the job market. This technology will bring its share of benefits, but also its share of challenges. With this in mind, it is crucial to understand, evaluate, and circumscribe this technology to optimise its positive effects while also mitigating its adverse effects. This is no easy task, since agents, especially generative agents, are not deterministic, and their quantitative evaluation poses a challenge. This paper presents a geopolitical simulator used as a testbed for the development, evaluation, and circumscription of Large Language Model (LLM) based intelligent agents. Using a fictitious world map of 20 countries enables agents and their hierarchies to interact, communicate, negotiate, and collaborate to achieve their goals. Agents’ behaviours and decisions are observed, evaluated, and quantified using methods that implement metrics to describe their level of ethics, collaboration, negotiation, creativity, and so on. The simulation is carried out using a hierarchy of classes, instantiated as objects, whose different variables have a causal and cascading impact on one another. Eventually, the various agents, based on popular LLMs (Chat GPT, Claude, Mistral, and others), will be compared and classified according to each of the metrics, providing the community with more in-depth knowledge of each of the major models.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Evaluating Agentic AI Through a Geopolitical Simulation Sandbox

  • Jean-Sébastien Dessureault,
  • Alain Thierry Iliho Manzi,
  • Soukaina Alaoui Ismaili,
  • Alitiana Mijoro Barisoa,
  • Donald Yankam Djioke,
  • Alex Bergeron,
  • Assefa Yared-Amare,
  • Mireille Lalancette,
  • Éric Bélanger

摘要

Intelligent agents are the technology of the moment, with their ability to interact in their environment according to the objectives set for them. Their immense potential makes them a disruptive technology, and they will undoubtedly have a global impact on the job market. This technology will bring its share of benefits, but also its share of challenges. With this in mind, it is crucial to understand, evaluate, and circumscribe this technology to optimise its positive effects while also mitigating its adverse effects. This is no easy task, since agents, especially generative agents, are not deterministic, and their quantitative evaluation poses a challenge. This paper presents a geopolitical simulator used as a testbed for the development, evaluation, and circumscription of Large Language Model (LLM) based intelligent agents. Using a fictitious world map of 20 countries enables agents and their hierarchies to interact, communicate, negotiate, and collaborate to achieve their goals. Agents’ behaviours and decisions are observed, evaluated, and quantified using methods that implement metrics to describe their level of ethics, collaboration, negotiation, creativity, and so on. The simulation is carried out using a hierarchy of classes, instantiated as objects, whose different variables have a causal and cascading impact on one another. Eventually, the various agents, based on popular LLMs (Chat GPT, Claude, Mistral, and others), will be compared and classified according to each of the metrics, providing the community with more in-depth knowledge of each of the major models.