Using Large Language Models to Probe Cognitive Constructs, Augment Data, and Design Instructional Materials

Significant advances in AI research, particularly in the field of machine learning, have opened up the possibility of solving problems in various specialist areas with the help of AI systems. In particular, advances with large language models have implications for education. This paper uses the currently most prominent large language model, GPT-4, to investigate the extent to which the process for solving physics problems can be modelled according to established problem-solving models. Furthermore, we evaluate in which ways specific prompting can be used to apply different problem solving strategies and which types of physical problems this large language model is able to generate when specifically prompted. Our investigations show that while GPT-4 is capable of solving physics problems, it occasionally uses common misconceptions and incomplete problem solving approaches. Specific problem solving strategies can be applied by giving specific instructions (i.e., prompts). Our findings have implications for utilizing GPT-4 for fostering problem solving strategies for students in physics.