Comparative Assessment of ChatGPT, DeepSeek, and Human Reviewers for Full-Text Screening in Systematic Reviews on the Impact of Air Pollution in Respiratory Diseases
摘要
Large language models (LLMs) are increasingly used in scientific research, education, and healthcare. Their roles in data searching, screening, extraction, and quality assessment hold promise for improving the systematic review process. However, concerns remain about their accuracy in literature research. This study aimed to compare the accuracy of LLMs and human reviewers in full-text screening for a systematic review. We searched for relevant studies from databases and registers. Full-text screening was performed by human reviewers, ChatGPT, and DeepSeek based on predefined inclusion and exclusion criteria. We observed that ChatGPT had 73.6% agreement with human reviewers (κ = 0.43), while DeepSeek had 70.3% (κ = 0.35). Moreover, ChatGPT showed 73.6% accuracy, high sensitivity (0.923), but low specificity (0.487) compared to the human consensus. Similarly, DeepSeek had 70.3% accuracy, higher sensitivity (0.962), but lower specificity (0.359). Both ChatGPT and DeepSeek show promise for assisting full-text screening in systematic reviews but require further evaluation with well-defined prompt engineering.