Deploying ChatGPT for Automated Tagging of Greek Dialogue Data of University Students

In this study, we propose a methodology of automated labelling for dialogues. We analyze dialogue data by deploying ChatGPT 3.5 model. Large language models (LLMs) have recently been in the focus of the scientific world and public since the release of ChatGPT. The approaches we used in this study were zero-shot, one-shot and few-shot learning. The model was asked to assign a label to each turn of dialogue using a tweaked version of the Issue-Based Information System (IBIS), and its labels were compared to those assigned by human raters, to evaluate its performance. Several different versions of prompts were used to investigate their effect on the model's performance, and it was also investigated whether the number of different labels affected the model's predictions. The results provide evidence that LLMs are not still able to fully process highly contextualized human dialogues but can provide worthwhile results if the assigned task is simplified. Also, we discuss on the reliability of labels created by a model, compared to labels created by a human annotator.