Scholarlink

Purpose <p>To evaluate the capability of large language models (LLM), specifically GPT-4 and o1, in assessing adherence to the MI-CLEAR-LLM checklist in previously published studies.</p> Materials and methods <p>A total of 159 medical research articles related to LLM applications were analyzed. Two models—GPT-4o and o1—were tested in both text-based and image-based modalities. Structured prompts incorporating reasoning strategies such as chain-of-thought and few-shot learning were used to extract information corresponding to the six core items of the MI-CLEAR-LLM checklist. Human evaluations from a prior study served as the reference standard. Each model was evaluated across three independent trials to assess consistency. Accuracy and inter-trial agreement were calculated for each checklist item.</p> Results <p>Both GPT-4o and o1 demonstrated high accuracy in extracting objective, explicitly reported items, such as LLM specifications (name, manufacturer, web access, 85.9–100%) and stochasticity parameters (63.6–95%). However, performance declined for context-dependent items, including prompt session handling (Item4, 51.5–70.7%) and test data independence (Item6, 59.6–76.8%). Text-based models generally showed superior inter-trial consistency, with GPT-4o-text achieving the highest Fleiss’ kappa (κ = 0.926). In contrast, image-based models exhibited greater variability (κ = 0.402–0.772).</p> Conclusion <p>LLMs show strong potential for automating the evaluation of reporting quality in medical research, particularly for clearly structured content. However, they still face substantial challenges in extracting context-dependent or inferential information.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Evaluating guideline adherence in LLM studies using LLMs

  • 期刊论文

  • 发表日期：

    2026年2月5日

查看全文
  • Ji Su Ko,
  • Hwon Heo,
  • Chong Hyun Suh,
  • Jeho Yi,
  • Woo Hyun Shim

摘要

Purpose

To evaluate the capability of large language models (LLM), specifically GPT-4 and o1, in assessing adherence to the MI-CLEAR-LLM checklist in previously published studies.

Materials and methods

A total of 159 medical research articles related to LLM applications were analyzed. Two models—GPT-4o and o1—were tested in both text-based and image-based modalities. Structured prompts incorporating reasoning strategies such as chain-of-thought and few-shot learning were used to extract information corresponding to the six core items of the MI-CLEAR-LLM checklist. Human evaluations from a prior study served as the reference standard. Each model was evaluated across three independent trials to assess consistency. Accuracy and inter-trial agreement were calculated for each checklist item.

Results

Both GPT-4o and o1 demonstrated high accuracy in extracting objective, explicitly reported items, such as LLM specifications (name, manufacturer, web access, 85.9–100%) and stochasticity parameters (63.6–95%). However, performance declined for context-dependent items, including prompt session handling (Item4, 51.5–70.7%) and test data independence (Item6, 59.6–76.8%). Text-based models generally showed superior inter-trial consistency, with GPT-4o-text achieving the highest Fleiss’ kappa (κ = 0.926). In contrast, image-based models exhibited greater variability (κ = 0.402–0.772).

Conclusion

LLMs show strong potential for automating the evaluation of reporting quality in medical research, particularly for clearly structured content. However, they still face substantial challenges in extracting context-dependent or inferential information.