This paper develops a formal framework to study deception in agentic AI systems, particularly those powered by large language models (LLMs). We conceptualize deception not as a local defect of individual agents, but as an emergent property of strategic interaction in multi-agent environments. To model this, we introduce the Generative-Conjectural LLM Equilibrium (GCLE), which captures the mutual reasoning dynamics between sender and receiver agents. Each agent operates with incomplete information and builds internal models of the other’s behavior through iterative, LLM-driven inference. This recursive reasoning structure leads to self-consistent yet potentially manipulable equilibria, revealing how misinformation can propagate through message-reasoning feedback loops. We further characterize conditions under which strategic deception becomes successful through a finite-horizon control formulation that guides receiver beliefs toward a desired behavior profile. Our analysis culminates in a spearphishing case study that illustrates how prompt design and cognitive modeling can be used to construct convincing deceptive messages. The results highlight the security risks posed by generative AI and the importance of principled reasoning models in understanding and mitigating deception in intelligent systems.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Generative-Conjectural LLM Equilibrium for Agentic AI Deception with Applications to Spearphishing

  • Quanyan Zhu

摘要

This paper develops a formal framework to study deception in agentic AI systems, particularly those powered by large language models (LLMs). We conceptualize deception not as a local defect of individual agents, but as an emergent property of strategic interaction in multi-agent environments. To model this, we introduce the Generative-Conjectural LLM Equilibrium (GCLE), which captures the mutual reasoning dynamics between sender and receiver agents. Each agent operates with incomplete information and builds internal models of the other’s behavior through iterative, LLM-driven inference. This recursive reasoning structure leads to self-consistent yet potentially manipulable equilibria, revealing how misinformation can propagate through message-reasoning feedback loops. We further characterize conditions under which strategic deception becomes successful through a finite-horizon control formulation that guides receiver beliefs toward a desired behavior profile. Our analysis culminates in a spearphishing case study that illustrates how prompt design and cognitive modeling can be used to construct convincing deceptive messages. The results highlight the security risks posed by generative AI and the importance of principled reasoning models in understanding and mitigating deception in intelligent systems.