<p>This paper examines the metaphysical and technical possibilities of superintelligent artificial agents and their potential to pose existential risks. While Nick Bostrom’s Instrumental Convergence Thesis (ICT) suggests that advanced intelligence will systematically adopt power-seeking subgoals, I argue that such risks are less imminent than often portrayed. By distinguishing between metaphysical and technical possibility, I highlight computational constraints such as combinatorial explosion and Moravec’s Paradox that limit superintelligence in practice. I further contend that because AI is built “by humans, for humans, about humans”, its motivations are likely to remain human-centered in the foreseeable future. Nevertheless, ICT underscores the importance of addressing the conditions under which intelligence and motivation combine to generate risk. I conclude by considering strategies for mitigation, including multi-agent architectures, modular service frameworks, and systems designed with uncertainty about human preferences. Overall, the existential threat of power-seeking superintelligence should be treated as possible but improbable in the short term, warranting caution without succumbing to alarmism.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Superintelligence, instrumental convergence, and the limits of AI apocalypse

  • Zachary Deutsch

摘要

This paper examines the metaphysical and technical possibilities of superintelligent artificial agents and their potential to pose existential risks. While Nick Bostrom’s Instrumental Convergence Thesis (ICT) suggests that advanced intelligence will systematically adopt power-seeking subgoals, I argue that such risks are less imminent than often portrayed. By distinguishing between metaphysical and technical possibility, I highlight computational constraints such as combinatorial explosion and Moravec’s Paradox that limit superintelligence in practice. I further contend that because AI is built “by humans, for humans, about humans”, its motivations are likely to remain human-centered in the foreseeable future. Nevertheless, ICT underscores the importance of addressing the conditions under which intelligence and motivation combine to generate risk. I conclude by considering strategies for mitigation, including multi-agent architectures, modular service frameworks, and systems designed with uncertainty about human preferences. Overall, the existential threat of power-seeking superintelligence should be treated as possible but improbable in the short term, warranting caution without succumbing to alarmism.