Scholarlink

<p>Thus far, the&#xa0;usage of generative artificial intelligence (GAI) has mainly been explored for content-based evaluations. Research quality evaluations and studies focusing on fields in the social sciences are needed. Therefore, we performed a quality evaluation using state-of-the-art GAI models to assess their usability in this respect. GPT-4o and DeepSeek-V3 were employed to select papers with very high quality from a set of high quality papers. Comparisons with human expert decisions, citation counts, and download counts were the basis for our quality assessments. GAI models can to some extent assess the academic value of high-quality papers. We found a positive but limited correlation between GAI-based quality scores with citation and download counts. We observed selective preferences of GAI’s quality assessment in favor of papers that use overt technical terminology, methodological frameworks, or seemingly broad but objectively worded expressions. GAI comparatively undervalues the quality of papers with high theoretical abstraction or strong socio-contextual embedding. Our study provides new empirical foundations for defining GAI’s role in research quality evaluation in the social sciences. Crucial guidance for academic institutions, libraries, funding agencies, and research management organizations is extracted. We expect that GAI trained to assess academic quality would produce classic paper selections in higher agreement with human experts than currently available models.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Can generative AI effectively perform quality evaluation within social sciences? A case study in library and information science

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年2月19日

查看全文
  • Yu Zhu,
  • Robin Haunschild,
  • Jiyuan Ye,
  • Huan Xie

摘要

Thus far, the usage of generative artificial intelligence (GAI) has mainly been explored for content-based evaluations. Research quality evaluations and studies focusing on fields in the social sciences are needed. Therefore, we performed a quality evaluation using state-of-the-art GAI models to assess their usability in this respect. GPT-4o and DeepSeek-V3 were employed to select papers with very high quality from a set of high quality papers. Comparisons with human expert decisions, citation counts, and download counts were the basis for our quality assessments. GAI models can to some extent assess the academic value of high-quality papers. We found a positive but limited correlation between GAI-based quality scores with citation and download counts. We observed selective preferences of GAI’s quality assessment in favor of papers that use overt technical terminology, methodological frameworks, or seemingly broad but objectively worded expressions. GAI comparatively undervalues the quality of papers with high theoretical abstraction or strong socio-contextual embedding. Our study provides new empirical foundations for defining GAI’s role in research quality evaluation in the social sciences. Crucial guidance for academic institutions, libraries, funding agencies, and research management organizations is extracted. We expect that GAI trained to assess academic quality would produce classic paper selections in higher agreement with human experts than currently available models.