Scholarlink

Background <p>Multimodal large language models (LLMs) are increasingly adopted in occupational therapy (OT) education, offering new opportunities for learning and assessment. However, their ability to handle professional licensing examinations remains unexplored. This study aimed to provide educational insights for OT educators by analyzing how multimodal LLMs perform on the Japanese National Examination for Occupational Therapists (JNEOT), particularly focused on visually-based questions.</p> Methods <p>The complete 199-item 2025 JNEOT (consisting of 177 text-only questions and 22 visually-based questions) was presented to OpenAI o3, Gemini 2.5, GPT-4.5, and Claude 3.7 in a zero-shot manner. Official answer keys served as the reference standard. Overall correct response rates and subscores were calculated for text-only versus visually-based and general versus practical items. We examined differences using Cochran’s Q, and significant omnibus results were followed by Bonferroni-corrected McNemar tests.</p> Results <p>All models achieved passing-level scores (82.4–91.0%) on text-based questions but showed a marked decline on visually based items (50.0-68.2%). No significant inter-model differences were found for visual tasks. Ten OT-specific questions were answered incorrectly by all models.</p> Conclusions <p>These findings suggest that LLMs can effectively reinforce foundational OT knowledge but remain limited in visual interpretation and contextual reasoning. For educators, the results offer practical guidance on using LLMs to enhance theoretical learning while maintaining OT educator-guided approaches for reflective and experiential competence development.</p> Trial registration <p>Not applicable.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

What the performance of multimodal LLMs on a national licensing exam teaches us about occupational therapy education

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年1月2日

查看全文
  • Kazuya Saita,
  • Yuichi Mine,
  • Satoru Amano

摘要

Background

Multimodal large language models (LLMs) are increasingly adopted in occupational therapy (OT) education, offering new opportunities for learning and assessment. However, their ability to handle professional licensing examinations remains unexplored. This study aimed to provide educational insights for OT educators by analyzing how multimodal LLMs perform on the Japanese National Examination for Occupational Therapists (JNEOT), particularly focused on visually-based questions.

Methods

The complete 199-item 2025 JNEOT (consisting of 177 text-only questions and 22 visually-based questions) was presented to OpenAI o3, Gemini 2.5, GPT-4.5, and Claude 3.7 in a zero-shot manner. Official answer keys served as the reference standard. Overall correct response rates and subscores were calculated for text-only versus visually-based and general versus practical items. We examined differences using Cochran’s Q, and significant omnibus results were followed by Bonferroni-corrected McNemar tests.

Results

All models achieved passing-level scores (82.4–91.0%) on text-based questions but showed a marked decline on visually based items (50.0-68.2%). No significant inter-model differences were found for visual tasks. Ten OT-specific questions were answered incorrectly by all models.

Conclusions

These findings suggest that LLMs can effectively reinforce foundational OT knowledge but remain limited in visual interpretation and contextual reasoning. For educators, the results offer practical guidance on using LLMs to enhance theoretical learning while maintaining OT educator-guided approaches for reflective and experiential competence development.

Trial registration

Not applicable.