The process of collecting rehabilitation material for speech disorders can be resource intensive, consume a lot of time and effort, and can also constrain the possibilities of customization. Improvements in multimodal generative Machine Learning can help accelerate and enhance the efficiency of this process. In this paper, we explore the use of such models to create rehabilitation material for aphasia. We propose a novel pipeline that uses existing rehabilitation concepts to generate not only therapy tasks using Large Language Models (LLMs), and images using Text-to-Image (T2I) models but also generate prompts for creating images and the therapy tasks themselves. We observe that combinations of Qwen (LLM) with both Stable Diffusion Turbo and Flux (both T2Is) tend to generate images with higher T2I evaluation metrics. In addition, we also investigate using Large Language and Vision Assistant (LLaVa) as a visual question answering tool to supplement LLM generated answers. Our experiments indicate that the answers generated by LLaVa have a higher alignment with Qwen (average alignment of 0.751 and 0.782 for inputs from SDXL+Qwen and Flux+Qwen, respectively) compared to Llama (average alignment of 0.721 and 0.740 for inputs from SDXL+Llama and Flux+Llama, respectively).

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Using Multi-modal Generative Models for Creating Speech Therapy Material

  • Mihir Mulye,
  • Stefan Conrad,
  • Stefan Knecht

摘要

The process of collecting rehabilitation material for speech disorders can be resource intensive, consume a lot of time and effort, and can also constrain the possibilities of customization. Improvements in multimodal generative Machine Learning can help accelerate and enhance the efficiency of this process. In this paper, we explore the use of such models to create rehabilitation material for aphasia. We propose a novel pipeline that uses existing rehabilitation concepts to generate not only therapy tasks using Large Language Models (LLMs), and images using Text-to-Image (T2I) models but also generate prompts for creating images and the therapy tasks themselves. We observe that combinations of Qwen (LLM) with both Stable Diffusion Turbo and Flux (both T2Is) tend to generate images with higher T2I evaluation metrics. In addition, we also investigate using Large Language and Vision Assistant (LLaVa) as a visual question answering tool to supplement LLM generated answers. Our experiments indicate that the answers generated by LLaVa have a higher alignment with Qwen (average alignment of 0.751 and 0.782 for inputs from SDXL+Qwen and Flux+Qwen, respectively) compared to Llama (average alignment of 0.721 and 0.740 for inputs from SDXL+Llama and Flux+Llama, respectively).