Causes of Failure

An important part of being able to deploy systems safely, is understanding when it is not safe to do so. To know when using a system is unsafe, one must understand why systems fail. To this end, this chapter presents a typology of different reasons that may lead autonomous weapon systems (AWS) to fail in the field: being cognisant of these factors enables AWS-users to predict whether there is increased risk of their systems failing after deployment. The chapter starts with distinguishing classical failures (human error and mechanical problems) from causes which are uniquely applicable to AI-systems. The chapter then discusses different possible causes of AI failure in sequence: training data issues, input data issues, out-of-distribution, proxy mistakes and bias. From this analysis, it is found that commanders play a particularly important role in predicting and addressing two causes of failure, input data issues and drift, as these arise from the particular context in which the system will be deployed. In contrast, commanders must be sensitive and knowledgeable about other causes of failure, but these are primarily for the Providing Entity to address. Finally, the chapter also discusses the concept of system failures, which holds that in some conditions, a convergence of circumstance occurs which produces a failure, but which cannot be predicted or prevented by definition. It is argued that both for prevention and post-hoc adjudication, system failure theory must be retained to avoid scapegoating.