To clarify how artificial intelligence models for image classification obtain their predictions, instance-based explanations give examples of other predictions obtained by those models for similar images. XAI (eXplainable Artificial Intelligence) techniques find those examples by getting images similar to the query using similarity metrics that consider image features. In this work, we introduce a Case-Based Reasoning approach that obtains these types of explanations (explanations by examples and counterfactuals) using Visual Question Answering techniques that take advantage of the high performance achieved by Large Language Models. We have evaluated our approach through an offline evaluation, in a context where images to classify and explain may be low quality. We obtained promising results and confirmed the benefits of using multimodal models in XAI, combining knowledge focused on images and natural language.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Visual Question Answering to Generate Case-Based Explanations for Image Classification

  • Ángel Bastardo-Rojas,
  • Marta Caro-Martínez

摘要

To clarify how artificial intelligence models for image classification obtain their predictions, instance-based explanations give examples of other predictions obtained by those models for similar images. XAI (eXplainable Artificial Intelligence) techniques find those examples by getting images similar to the query using similarity metrics that consider image features. In this work, we introduce a Case-Based Reasoning approach that obtains these types of explanations (explanations by examples and counterfactuals) using Visual Question Answering techniques that take advantage of the high performance achieved by Large Language Models. We have evaluated our approach through an offline evaluation, in a context where images to classify and explain may be low quality. We obtained promising results and confirmed the benefits of using multimodal models in XAI, combining knowledge focused on images and natural language.