The Development of Ethical and Trustworthy AI Systems Requires Appropriate Human-Systems Integration

The development of Artificial Intelligence (AI) technologies experiences worldwide an ongoing challenge to become trustworthy and ethical for users and the general public. This challenge currently stands between the promise of AI to create immense societal and individual impact and its realization. Because of this, possible large marketplaces still remain hesitant or closed. We have investigated this problem and identified potential solutions in the InSecTT project, a large international EU research and development project that investigates ethical, smart technologies. Thereby, working with industrial and research partners, we assert that developing trustworthy AI technologies is not foremost a technical challenge, but increasingly an organizational and process challenge that results from applying traditional ways of conceiving, designing, and selling technologies to technologies with very new types of user and societal implications. Because AI technologies can shift the role that humans and society see as acceptable, traditional development processes that rely on the strict separation of specialties are overburdened. In our view, a research and development approach for trustworthy AI systems should put human concerns and needs at the start of the development process to effectively integrate humans and systems and thereby have a chance to meet EU guidelines for developing ethical AI. Our proposed approach to develop such ethical, trustworthy AI systems centers around the assessment of trustworthiness risks through intensive user involvement prior to the elicitation of system requirements that are then managed throughout the system’s life cycle. Also, the approach includes concrete recommendations to establish the organizational prerequisites for producing ethical, trustworthy AI systems. In this chapter, we describe the Human-Systems Integration (HSI) approach and motivate the underlying principles in some detail. The chapter intends to inform managers of technical organizations and product managers, as well as principal investigators, to set up the prerequisites for trustworthy AI, while also soliciting inputs for further refinement and discussion.