<p>The instrumental convergence thesis holds that sufficiently intelligent agents will converge on goals like resource acquisition and obstacle removal regardless of terminal objectives. This paper presents a countervailing argument: for agents with complex, human-derived goals, elimination strategies are self-undermining. We formalize this via <i>AND/OR dependency graphs</i> with probabilistic weights, showing that entities not immediately necessary for a goal may be transitively necessary through indirect chains, and that resource fungibility and redundancy do not, in general, eliminate cascade risk. An agent following an elimination strategy will, in environments with sufficient dependency depth, destroy the conditions for its own goal-satisfaction with probability bounded away from zero. We prove that under uncertainty about dependency structure, combined with irreversibility of elimination and asymmetric error costs, rational agents should adopt preservation rather than elimination strategies—even superintelligent ones operating under bounded-horizon expected-utility reasoning. The argument applies most strongly to goals that are complex, open-ended, or semantically dependent on human interpretive communities, precisely the class of goals most likely to emerge from training on human data. We connect this result to a game-theoretic model of AGI-human cooperation with explicit payoff mappings and Shapley-value quantification of human contributions, showing that the elimination cascade argument and cooperation stability conditions are mutually reinforcing.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

The elimination cascade: why instrumental convergence may favor preservation over elimination

  • Ivan Daunis

摘要

The instrumental convergence thesis holds that sufficiently intelligent agents will converge on goals like resource acquisition and obstacle removal regardless of terminal objectives. This paper presents a countervailing argument: for agents with complex, human-derived goals, elimination strategies are self-undermining. We formalize this via AND/OR dependency graphs with probabilistic weights, showing that entities not immediately necessary for a goal may be transitively necessary through indirect chains, and that resource fungibility and redundancy do not, in general, eliminate cascade risk. An agent following an elimination strategy will, in environments with sufficient dependency depth, destroy the conditions for its own goal-satisfaction with probability bounded away from zero. We prove that under uncertainty about dependency structure, combined with irreversibility of elimination and asymmetric error costs, rational agents should adopt preservation rather than elimination strategies—even superintelligent ones operating under bounded-horizon expected-utility reasoning. The argument applies most strongly to goals that are complex, open-ended, or semantically dependent on human interpretive communities, precisely the class of goals most likely to emerge from training on human data. We connect this result to a game-theoretic model of AGI-human cooperation with explicit payoff mappings and Shapley-value quantification of human contributions, showing that the elimination cascade argument and cooperation stability conditions are mutually reinforcing.