Manually and GPT-4 Created Feedback Questionnaires in Unmoderated Studies: A Comparative Case Study
摘要
Generative artificial intelligence (AI) tools like ChatGPT are increasingly integrated into personal and professional business domains, from facial recognition systems in smartphones to product development and talent recruitment. Despite their widespread use, the full potential of AI is often underutilized. Given the increasing importance of User Experience (UX) Research in software product and service design, this paper explores how AI can support User Research (UR) practitioners. Questionnaires are commonly used in UR to gather user feedback, but crafting effective questionnaires is time-consuming and cognitively demanding. This study evaluates the usefulness of GPT-4 in generating feedback questionnaires for unmoderated studies and discusses practical implementation at a global business software provider. Specifically, and as assessed by UX researchers, it compares the quality of GPT-4-generated questionnaires with manually created ones. Results indicate that participants perceived GPT-4-generated questionnaires as qualitatively inferior to manually created ones, although the GPT-4 questions demonstrated notable quality. This suggests that GPT-4 can serve as an assistant, potentially reducing the time required to create questionnaires or improving their quality. Given the rapid advancement of AI tools, further, continuous research with the latest AI models is necessary.