Recent advancements in Large Language Models (LLMs) have catalyzed a paradigm shift from static prediction systems to agentic AI—intelligent agents capable of reasoning, interacting with tools, and adapting to complex tasks. While LLM agentic systems have shown promise across many domains, their application to medical imaging remains in its infancy. In this work, we introduce AURA, the first visual linguistic explainability agent designed specifically for comprehensive analysis, explanation, and evaluation of medical images. By enabling dynamic interactions, contextual explanations, and hypothesis testing, AURA represents a significant advancement toward more transparent, adaptable, and clinically aligned AI systems. This work highlights the promise of agentic AI in transforming medical image analysis from static predictions to interactive decision support. Leveraging Qwen-32B, an LLM-based architecture, AURA integrates a modular toolbox comprising: (i) a segmentation suite with phase grounding, pathology segmentation, and anatomy segmentation to localize clinically meaningful regions; (ii) a counterfactual image generation module that supports reasoning through image-level explanations; and (iii) a set of evaluation tools, including pixel-wise difference map analysis, classification, and advanced state-of-the-art components, to assess the diagnostic relevance and visual interpretability of the results. Our code is accessible through the project website ( https://nimafathi.github.io/AURA/ ).

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

AURA: A Multi-modal Medical Agent for Understanding, Reasoning and Annotation

  • Nima Fathi,
  • Amar Kumar,
  • Tal Arbel

摘要

Recent advancements in Large Language Models (LLMs) have catalyzed a paradigm shift from static prediction systems to agentic AI—intelligent agents capable of reasoning, interacting with tools, and adapting to complex tasks. While LLM agentic systems have shown promise across many domains, their application to medical imaging remains in its infancy. In this work, we introduce AURA, the first visual linguistic explainability agent designed specifically for comprehensive analysis, explanation, and evaluation of medical images. By enabling dynamic interactions, contextual explanations, and hypothesis testing, AURA represents a significant advancement toward more transparent, adaptable, and clinically aligned AI systems. This work highlights the promise of agentic AI in transforming medical image analysis from static predictions to interactive decision support. Leveraging Qwen-32B, an LLM-based architecture, AURA integrates a modular toolbox comprising: (i) a segmentation suite with phase grounding, pathology segmentation, and anatomy segmentation to localize clinically meaningful regions; (ii) a counterfactual image generation module that supports reasoning through image-level explanations; and (iii) a set of evaluation tools, including pixel-wise difference map analysis, classification, and advanced state-of-the-art components, to assess the diagnostic relevance and visual interpretability of the results. Our code is accessible through the project website ( https://nimafathi.github.io/AURA/ ).