Scholarlink

Computational Thinking (CT) is essential for K-12 learners. Yet, educators lack methods and tools to assess their students. We present a pilot study exploring how large language models can automate CT evidence extraction – states, transitions, connections, and cycles – from embodied games represented as Finite State Automata (FSA). We built a dataset from the platform’s logs to evaluate four large language models: three open-source (Llama 3.2, Tulu, DeepSeek’s R1) and OpenAI’s o1. Initial prompts caused models to misinterpret the word “loops” as self-loops. Revising terminology to “cycles” improved accuracy. Quantitative results showed OpenAI’s o1 and DeepSeek’s R1 achieved perfect F1 scores (1.0) in identifying CT evidence, while Llama 3.2 (0.974) and Tulu (0.961) performed competitively. Informal interviews with two educators as validation highlighted practical needs: tools should map CT evidence to learning standards and track progress (e.g., flagging missed concepts). However, challenges include models’ occasional fabrication of details, lack of empathetic responses, and reliance on technical jargon. Future directions include longitudinal progress analysis, multimodal inputs, and prioritizing insights for feedback.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Toward Extracting Computational Thinking Evidence with Large Language Models: Empowering K-12 Educators

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • William Lee,
  • Ivon Arroyo

摘要

Computational Thinking (CT) is essential for K-12 learners. Yet, educators lack methods and tools to assess their students. We present a pilot study exploring how large language models can automate CT evidence extraction – states, transitions, connections, and cycles – from embodied games represented as Finite State Automata (FSA). We built a dataset from the platform’s logs to evaluate four large language models: three open-source (Llama 3.2, Tulu, DeepSeek’s R1) and OpenAI’s o1. Initial prompts caused models to misinterpret the word “loops” as self-loops. Revising terminology to “cycles” improved accuracy. Quantitative results showed OpenAI’s o1 and DeepSeek’s R1 achieved perfect F1 scores (1.0) in identifying CT evidence, while Llama 3.2 (0.974) and Tulu (0.961) performed competitively. Informal interviews with two educators as validation highlighted practical needs: tools should map CT evidence to learning standards and track progress (e.g., flagging missed concepts). However, challenges include models’ occasional fabrication of details, lack of empathetic responses, and reliance on technical jargon. Future directions include longitudinal progress analysis, multimodal inputs, and prioritizing insights for feedback.