Scholarlink

<p>The course project report (CPR) is a crucial component for assessing students’ learning outcomes from courses they are studying. It assesses practical skills, academic writing, and logical thinking. In recent times, researchers have increasingly leveraged large language models (LLMs) to promote automated essay scoring (AES) in the education intelligence field due to its strong generalization and reasoning abilities. However, the existing LLM-based AES method design is based solely on writing proficiency and inevitably ignores the importance of assessment of cognitive engagement and practical competencies in CPRs. Additionally, CPR writing is a reflective process that includes knowledge-inquiry and cognition through critical thinking (CT), which have rarely been explored in the design of prompts for specific LLMs. To tackle this issue, we propose a novel, guided generative AI (GenAI) prompting framework for automated CPR assessment. It is created by integrating the Paul-Elder critical thinking concept into prompt design to enhance domain-specific knowledge transfer and the analytical capabilities of GenAI LLMs. Rather than focusing solely on language structure or writing skills, our approach emphasizes critical thinking evaluation using the Paul-Elder CT framework. Specifically, our framework—PEG-Prompt—evaluates CPR across six dimensions—structure, logic, coherence, originality, citation, and knowledge proficiency—to evaluate CPRs comprehensively from the aspects of practical competencies, analytical reasoning, and writing skills. To further enhance the CPR assessment performance of PEG-Prompt, we combine PEG-Prompt with extracted key content from reports and representative examples of few-shot scoring. Experimental results demonstrate that PEG-Prompt significantly improves the correlation between LLM-generated scores and human scores. The enhanced framework may enable students to receive helpful feedback and summaries of their CPR results through GenAI once it has been calibrated with human evaluators.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Evaluating the Efficacy of a Multifaceted Prompt for Use with LLMs to Evaluate Course Project Reports

  • 期刊论文

  • 发表日期：

    2026年4月23日

查看全文
  • Qingyang Sun,
  • Jialu Zhang,
  • Peng Sheng,
  • Qianyi Wang,
  • Tianrui Wang,
  • Heng Li,
  • Hanshu Zhan,
  • Xiaoqing Zhang,
  • Jiang Liu

摘要

The course project report (CPR) is a crucial component for assessing students’ learning outcomes from courses they are studying. It assesses practical skills, academic writing, and logical thinking. In recent times, researchers have increasingly leveraged large language models (LLMs) to promote automated essay scoring (AES) in the education intelligence field due to its strong generalization and reasoning abilities. However, the existing LLM-based AES method design is based solely on writing proficiency and inevitably ignores the importance of assessment of cognitive engagement and practical competencies in CPRs. Additionally, CPR writing is a reflective process that includes knowledge-inquiry and cognition through critical thinking (CT), which have rarely been explored in the design of prompts for specific LLMs. To tackle this issue, we propose a novel, guided generative AI (GenAI) prompting framework for automated CPR assessment. It is created by integrating the Paul-Elder critical thinking concept into prompt design to enhance domain-specific knowledge transfer and the analytical capabilities of GenAI LLMs. Rather than focusing solely on language structure or writing skills, our approach emphasizes critical thinking evaluation using the Paul-Elder CT framework. Specifically, our framework—PEG-Prompt—evaluates CPR across six dimensions—structure, logic, coherence, originality, citation, and knowledge proficiency—to evaluate CPRs comprehensively from the aspects of practical competencies, analytical reasoning, and writing skills. To further enhance the CPR assessment performance of PEG-Prompt, we combine PEG-Prompt with extracted key content from reports and representative examples of few-shot scoring. Experimental results demonstrate that PEG-Prompt significantly improves the correlation between LLM-generated scores and human scores. The enhanced framework may enable students to receive helpful feedback and summaries of their CPR results through GenAI once it has been calibrated with human evaluators.