<p>Contemporary intelligent systems increasingly operate with high levels of autonomy, remaining formally accurate, calibrated and compliant with established performance standards. However, recent failures in large-scale sociotechnical systems suggest that harm can arise even in the absence of an explicit malfunction. This article introduces the concept of autonomous risk to describe how intelligent systems can become structurally dangerous without failing in conventional technical terms. A formal operationalization of autonomous risk is proposed as a dynamic interaction between autonomy, opacity, supervision and instability, distinguishing the baseline risk estimate from its endogenous amplification under autonomy. The proposed formulation is model-agnostic, allowing autonomous risk to be instantiated from any continuous signal of deviation from nominal system behavior, while preserving the same structural relationship between autonomy, opacity, supervision, and instability. Risk is modeled not as a discrete event or outcome, but as a path-dependent property of the system that evolves over time. Through a series of simulated decision environments, it is demonstrated that high risk is concentrated in specific regimes of partial autonomy, where systems are sufficiently independent to act, but insufficiently robust to self-correct. Empirical results reveal non-linear risk amplification, latent instability fields, and emerging schema-like trajectories that remain invisible to standard performance metrics. These findings challenge prevailing governance approaches focused on accuracy, explainability, and episodic oversight. It is argued that effective governance must shift to adaptive, trajectory-aware oversight mechanisms capable of detecting structural risks <i>ex ante</i>, before clear failure occurs. The framework contributes a measurable diagnostic lens to anticipate dangerous system dynamics in smart systems before collapse.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Autonomous risk: when intelligent systems become dangerous without failing

  • Erivelton Pinheiro de Menezes

摘要

Contemporary intelligent systems increasingly operate with high levels of autonomy, remaining formally accurate, calibrated and compliant with established performance standards. However, recent failures in large-scale sociotechnical systems suggest that harm can arise even in the absence of an explicit malfunction. This article introduces the concept of autonomous risk to describe how intelligent systems can become structurally dangerous without failing in conventional technical terms. A formal operationalization of autonomous risk is proposed as a dynamic interaction between autonomy, opacity, supervision and instability, distinguishing the baseline risk estimate from its endogenous amplification under autonomy. The proposed formulation is model-agnostic, allowing autonomous risk to be instantiated from any continuous signal of deviation from nominal system behavior, while preserving the same structural relationship between autonomy, opacity, supervision, and instability. Risk is modeled not as a discrete event or outcome, but as a path-dependent property of the system that evolves over time. Through a series of simulated decision environments, it is demonstrated that high risk is concentrated in specific regimes of partial autonomy, where systems are sufficiently independent to act, but insufficiently robust to self-correct. Empirical results reveal non-linear risk amplification, latent instability fields, and emerging schema-like trajectories that remain invisible to standard performance metrics. These findings challenge prevailing governance approaches focused on accuracy, explainability, and episodic oversight. It is argued that effective governance must shift to adaptive, trajectory-aware oversight mechanisms capable of detecting structural risks ex ante, before clear failure occurs. The framework contributes a measurable diagnostic lens to anticipate dangerous system dynamics in smart systems before collapse.