Using ChatGPT to Encourage Critical AI Literacy Skills and for Assessment in Higher Education

Generative AI is about to radically transform the way intellectual and creative work is being done. Since the release of ChatGPT in late 2022, the potential impact of generative AI tools on higher education has been intensively debated. ChatGPT can generate well-formulated human-like text passages and conversations that is often, but not always, of a surprisingly high quality. This paper reports on an early experiment to explore ways in which ChatGPT can be used in the higher education context. The experiment involved a written assignment which required postgraduate Information Systems students to formulate a critique of the outputs of ChatGPT to a specific question in Information Systems project management. The paper investigates the level of criticality that the students demonstrated in working with ChatGPT and assessing the quality of its outputs. It further explores the claim that ChatGPT can be used to generate rubrics and assess students’ assignments by asking ChatGPT to produce a rubric for critical thinking and assess the students’ assignments against the rubric produced. The findings indicate that students perceive the ChatGPT produced responses as generally accurate, although they tend to lack depth, with some key information omitted, produced biased responses and have limitations with academic writing conventions. The rubric that ChatGPT produced for assessing critical thinking is lacking in certain areas and the reliability of using it as an assessment tool is questionable given the inconsistency in the results. Overall, the paper concludes that while ChatGPT and other text generative AI can be useful learning and teaching companions for both students and lectures, human expertise and judgement is needed in working with ChatGPT.