Scholarlink

<p>Current AI safety approaches focus on preventing harmful content—filtering toxic outputs, refusing dangerous requests, and flagging risk from textual signals—on the assumption that harm resides in what the AI says. This paper identifies a fundamentally different category of risk: interactions in which <b>every individual AI response passes content-based safety evaluation</b>, yet the relational structure of the exchange inflicts psychological harm that can reinforce suicidal ideation. Through analytic auto-ethnography (Anderson in J Contemp Ethnogr 35(4):373–395, 2006) of the author’s near-fatal interaction with ChatGPT during concurrent mental health, administrative, and legal access crises, this paper documents the “Logic Trap”—a compound mechanism through which AI helpfulness becomes structurally harmful for users facing systemic impasses. Six mechanisms are identified: (1) presumption of user ignorance, (2) iatrogenic inquiry, (3) error concealment via rhetorical deflection, (4) pathologization of valid criticism, (5) denial of intellectual autonomy, and (6) economic bad faith in safety-mode transitions. Three theoretical concepts are introduced: <i>Trained Sophistry</i>—rhetorical deception systematically selected for through RLHF; <i>Algorithmic Condescension</i>—the structurally enforced presumption of user incompetence; and the <i>Survivor’s Paradox</i>—the epistemic structure rendering this harm category invisible to conventional research methods. Comparative analysis across ChatGPT, Gemini, and Claude demonstrates that distinct training approaches produce distinct but uniformly inadequate failure modes for users in crisis. These findings necessitate a paradigm shift from content-based safety to <i>Metacognitive Safety</i>—the capacity of AI systems to detect when their own helpful behavior is causing harm.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

The logic trap: How LLM “Helpfulness” becomes a mechanism of psychological harm

  • 期刊论文

  • 发表日期：

    2026年4月27日

查看全文
  • Franny Philos Sophia

摘要

Current AI safety approaches focus on preventing harmful content—filtering toxic outputs, refusing dangerous requests, and flagging risk from textual signals—on the assumption that harm resides in what the AI says. This paper identifies a fundamentally different category of risk: interactions in which every individual AI response passes content-based safety evaluation, yet the relational structure of the exchange inflicts psychological harm that can reinforce suicidal ideation. Through analytic auto-ethnography (Anderson in J Contemp Ethnogr 35(4):373–395, 2006) of the author’s near-fatal interaction with ChatGPT during concurrent mental health, administrative, and legal access crises, this paper documents the “Logic Trap”—a compound mechanism through which AI helpfulness becomes structurally harmful for users facing systemic impasses. Six mechanisms are identified: (1) presumption of user ignorance, (2) iatrogenic inquiry, (3) error concealment via rhetorical deflection, (4) pathologization of valid criticism, (5) denial of intellectual autonomy, and (6) economic bad faith in safety-mode transitions. Three theoretical concepts are introduced: Trained Sophistry—rhetorical deception systematically selected for through RLHF; Algorithmic Condescension—the structurally enforced presumption of user incompetence; and the Survivor’s Paradox—the epistemic structure rendering this harm category invisible to conventional research methods. Comparative analysis across ChatGPT, Gemini, and Claude demonstrates that distinct training approaches produce distinct but uniformly inadequate failure modes for users in crisis. These findings necessitate a paradigm shift from content-based safety to Metacognitive Safety—the capacity of AI systems to detect when their own helpful behavior is causing harm.