Evaluation, Monitoring, and Improvement

In this chapter, the principles for evaluating and monitoring artificial intelligence (AI) solutions in the radiology department are discussed. In particular, the integration of the principles of business intelligence (BI) is suggested as a crucial step for assessing the impact of AI on radiological operations. When establishing a BI framework, it is of importance to start with defining clear measurable objectives that align with the strategic goals of the department and institution. With clear measurable objectives in place, connecting relevant data sources to the framework will allow relevant key performance indicators (KPIs) to be determined. KPIs are quantifiable metrics that provide information on the impact of technological and organisational changes on radiological operations and can be used to measure the success of a project or initiative in achieving its strategic objectives or goals. It is likely that the use of multiple KPIs in conjunction will offer a holistic overview of the impact, providing a comprehensive understanding of the changes and their effects. This emphasises the need of establishing a logical relation between the defined objectives and KPIs to monitor the impact of AI technology. Therefore, given the complexity of many radiological workflows, where a completely AI-enhanced workflow will most probably consist of multiple AI solutions working together in an orchestrated fashion, having an appropriate performance monitoring system in place will assist department management and different stakeholders to carefully monitor the impact of multiple AI applications and make informed decisions to optimise clinical workflows.