Using Multi-modal Generative Models for Creating Speech Therapy Material
摘要
The process of collecting rehabilitation material for speech disorders can be resource intensive, consume a lot of time and effort, and can also constrain the possibilities of customization. Improvements in multimodal generative Machine Learning can help accelerate and enhance the efficiency of this process. In this paper, we explore the use of such models to create rehabilitation material for aphasia. We propose a novel pipeline that uses existing rehabilitation concepts to generate not only therapy tasks using Large Language Models (LLMs), and images using Text-to-Image (T2I) models but also generate prompts for creating images and the therapy tasks themselves. We observe that combinations of Qwen (LLM) with both Stable Diffusion Turbo and Flux (both T2Is) tend to generate images with higher T2I evaluation metrics. In addition, we also investigate using Large Language and Vision Assistant (LLaVa) as a visual question answering tool to supplement LLM generated answers. Our experiments indicate that the answers generated by LLaVa have a higher alignment with Qwen (average alignment of 0.751 and 0.782 for inputs from SDXL+Qwen and Flux+Qwen, respectively) compared to Llama (average alignment of 0.721 and 0.740 for inputs from SDXL+Llama and Flux+Llama, respectively).