Comparative Analysis of AI Chatbot Responses in a Computer-Aided Drawing Course
摘要
This study provides a comprehensive investigation into the potential of AI chatbots to assist civil engineering students in mastering the use of AutoCAD, a fundamental tool in the field. The research involved 26 civil engineering students who had completed a 16-week course on computer-aided technical drawing using AutoCAD. Each student developed a bank of 20 questions covering 11 key topics, which were then evaluated by seven prominent AI models, including GPT-3.5, GPT-4, Claude-3.5 Sonnet, Gemini 1.5 Pro, Perplexity, You, and Llama 3.1. The chatbots’ responses were scored on a three-point scale, and the results were meticulously analyzed to assess their performance across the different AutoCAD-related topics. The findings reveal that while the AI models demonstrated satisfactory overall scores, with most achieving above 70 points on a 100-point scale, their capabilities varied significantly across the topics. The chatbots exhibited stronger performance on more straightforward tasks, such as Array Functions, but struggled with complex, contextual inquiries related to areas like Help and Workspace and Plan Organization. These results highlight the promise of AI in supporting civil engineering education, while also underscoring the continued importance of human instructors in providing the critical thinking, creativity, and emotional intelligence required for effective learning. The study’s insights pave the way for future research on integrating AI tools into engineering curricula, leveraging their strengths while addressing their limitations, and exploring the ethical considerations of AI-assisted learning in highly specialized technical domains.