Integrating Multi-Modal Language Models and Machine Learning in Dermatology
摘要
Objective: To describe, implement, evaluate, and quantify a novel and adaptable Artificial Intelligence-empowered methodology aimed at supporting a dermatologist’s workflow in assessing and diagnosing skin conditions, leveraging AI’s deep image analytic power and reasoning. Patients and Methods: We employ large language, transformer-based vision models for image analysis, sophisticated machine learning tools for guideline-based segmentation, and measuring tasks in our system. As no single technology is sufficient on its own for efficient use by dermatologists, we apply a sequential logic with agency to improve outcomes. Results: Using natural language processing methods and incorporating human expert evaluation, our system achieved a weighted accuracy of 87% on the dataset used, demonstrating its reasoning and diagnostic capabilities. Conclusions: This study serves as a proof of concept for the application of AI in dermatology, highlighting its potential to enhance the patient journey for which we approximate the value of such interventions in healthcare using graph theory with an associated cost-optimisation objective function.