Optimizing ChatGPT-Assisted Evaluation of Vietnamese English Majors’ Research Proposals: A Tree-of-Thought and Rubric-Based Approach
摘要
This study examines the enhancement of ChatGPT-assisted evaluation in assessing the methodological components of Vietnamese undergraduate English majors’ research proposals at Saigon University, Vietnam, by using a more concise rubric and a Tree-of-thought (ToT) framework with a zero-shot learning approach. The evaluation focuses on the research titles, research questions, hypotheses, paradigm, design, and techniques of 40 research proposals. Compared to the detailed rubric with a chain-of-thought prompting style in Experiment 1, results indicate a significant improvement in ChatGPT’s intra-rater and inter-rater reliability, particularly in complex reasoning areas such as the justification and effectiveness of the research paradigm, design, and techniques. However, reliability declined in straightforward aspects, likely due to the inherent complexity of ToT. In addition, inconsistency emerged in evaluating research significance, which requires contextualized field knowledge. While these discoveries flag ToT’s limitations in evaluating simple aspects and those that require knowledge about the current literature, they also showcase its potential in enhancing ChatGPT’s performance in complex reasoning evaluations, rendering it a useful approach in assessing research methodology. Despite these findings, this experiment suffers from limited generalizability and an incomplete demonstration in evaluating research hypotheses, as this section was absent from all samples.