<p>The development of artificial general intelligence (AGI) is likely to be one of humanity’s most consequential technological advancements. Leading AI labs and scientists have called for the global prioritization of AI safety, citing existential risks comparable to nuclear war. However, research on catastrophic risks and AI alignment is often met with skepticism, even by machine learning experts. Furthermore, online debate over the existential risk of AI has begun to turn tribal (e.g. using labels such as “doomer” or “accelerationist”). Until now, no systematic study has explored the patterns of belief and the levels of familiarity with AI safety concepts among experts. I surveyed 111 AI experts (NeurIPS authors, ML-focused PhD students, and industry ML engineers; Sect.&#xa0;<InternalRef RefID="Sec4">2.2</InternalRef> outlines the selection criteria) on their familiarity with AI safety concepts, key objections to AI safety, and reactions to safety arguments. My findings reveal that AI experts cluster into two viewpoints—an “AI as controllable tool” and an “AI as uncontrollable agent” perspective—diverging in beliefs toward the importance of AI safety. While most experts (78%) agreed or strongly agreed that “technical AI researchers should be concerned about catastrophic risks”, many were unfamiliar with specific AI safety concepts. For example, only 21% of surveyed experts had heard of “instrumental convergence,” a fundamental concept in AI safety predicting that advanced AI systems will tend to pursue common sub-goals (such as self-preservation). The least concerned participants were the least familiar with concepts like this, suggesting that effective communication of AI safety should begin with establishing clear conceptual foundations in the field.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Why do experts disagree on existential risk? A survey of AI experts

  • Severin Field

摘要

The development of artificial general intelligence (AGI) is likely to be one of humanity’s most consequential technological advancements. Leading AI labs and scientists have called for the global prioritization of AI safety, citing existential risks comparable to nuclear war. However, research on catastrophic risks and AI alignment is often met with skepticism, even by machine learning experts. Furthermore, online debate over the existential risk of AI has begun to turn tribal (e.g. using labels such as “doomer” or “accelerationist”). Until now, no systematic study has explored the patterns of belief and the levels of familiarity with AI safety concepts among experts. I surveyed 111 AI experts (NeurIPS authors, ML-focused PhD students, and industry ML engineers; Sect. 2.2 outlines the selection criteria) on their familiarity with AI safety concepts, key objections to AI safety, and reactions to safety arguments. My findings reveal that AI experts cluster into two viewpoints—an “AI as controllable tool” and an “AI as uncontrollable agent” perspective—diverging in beliefs toward the importance of AI safety. While most experts (78%) agreed or strongly agreed that “technical AI researchers should be concerned about catastrophic risks”, many were unfamiliar with specific AI safety concepts. For example, only 21% of surveyed experts had heard of “instrumental convergence,” a fundamental concept in AI safety predicting that advanced AI systems will tend to pursue common sub-goals (such as self-preservation). The least concerned participants were the least familiar with concepts like this, suggesting that effective communication of AI safety should begin with establishing clear conceptual foundations in the field.