Scholarlink

OpenAI recently launched sophisticated Artificial Intelligence (AI) chatbots, namely ChatGPT and GPT-4, that can process user input and generate human-like responses. As AI technology continues to play an increasingly vital role in education, this study aimed to explore its potential application in facilitating the analysis of classroom discourse. Specifically, we evaluated the accuracy of ChatGPT and GPT-4 hosted on campus in coding classroom discourse data. The setting was a fully online, secondary school mathematics lesson in a Chinese classroom. The performance of the AI coders was compared with that of human coders. The results indicated substantial discrepancies between the human coders and ChatGPT (Cohen’s Kappa = 0.18), the human coders and GPT-4 (Cohen’s Kappa = 0.37), and ChatGPT and GPT-4 (Cohen’s Kappa = 0.21). Despite GPT-4’s superior performance over ChatGPT, the degree of agreement between each pair of coders could not be considered satisfactory. An in-depth analysis of their coding results revealed that ChatGPT and GPT-4 had limitations in fully comprehending the context, discerning the underlying meanings of text, and understanding certain context-specific issues. Nevertheless, their capacity to construct new and practical codes demonstrated their potential as valuable tools for assisting researchers in classroom discourse analysis.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

An Exploratory Study of Using AI Tools to Analyse Classroom Discourse Data

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Chung Kwan Lo,
  • Simin Xu,
  • Gaowei Chen

摘要

OpenAI recently launched sophisticated Artificial Intelligence (AI) chatbots, namely ChatGPT and GPT-4, that can process user input and generate human-like responses. As AI technology continues to play an increasingly vital role in education, this study aimed to explore its potential application in facilitating the analysis of classroom discourse. Specifically, we evaluated the accuracy of ChatGPT and GPT-4 hosted on campus in coding classroom discourse data. The setting was a fully online, secondary school mathematics lesson in a Chinese classroom. The performance of the AI coders was compared with that of human coders. The results indicated substantial discrepancies between the human coders and ChatGPT (Cohen’s Kappa = 0.18), the human coders and GPT-4 (Cohen’s Kappa = 0.37), and ChatGPT and GPT-4 (Cohen’s Kappa = 0.21). Despite GPT-4’s superior performance over ChatGPT, the degree of agreement between each pair of coders could not be considered satisfactory. An in-depth analysis of their coding results revealed that ChatGPT and GPT-4 had limitations in fully comprehending the context, discerning the underlying meanings of text, and understanding certain context-specific issues. Nevertheless, their capacity to construct new and practical codes demonstrated their potential as valuable tools for assisting researchers in classroom discourse analysis.