A Case Study on AI System Evaluation from Users’ Viewpoints

This paper proposes a process of AI system evaluation from users’ viewpoints, utilizing quality-in-use model adopted in ISO25029:2013. The proposed approach provides a method to map service requirement components of an AI service to quality-in-use components and sum up individual evaluation results to derive comprehensive results for three components, benefit, freedom from risk and adaptability. The authors applied the process to a use case of autonomous drone, which delivers luggage with autonomous control capability and have different stakeholder groups such as service provider, service user and residents of flight area. The case study showed that mapping of non-functional service requirement components (such as safety and other risk management) to quality-in-use components, and their comprehensive evaluation worked well, providing different evaluation results for each stakeholder. Also, it showed some refinement needed for the practical use of the process.