Thematic analysis of transcripts and qualitative data is frequently employed in design research and other fields like medicine. This analysis typically involves creating a coding scheme for a coder to classify data according to predefined codes. Using multiple coders mitigates individual biases, enhancing the reliability of the coding process and providing a measure of intercoder reliability, thus adding rigor to the research. In design research, intercoder reliability (ICR) is crucial in areas such as user experience research, qualitative data analysis, and content analysis of design artifacts. However, for extensive datasets, involving multiple coders can be resource-intensive. Hence, an AI Agent like GPT (Generative Pre-trained Transformers) could act as a coder, reducing the cost and effort of evaluating intercoder reliability. This study explores ChatGPT’s role as a coder alongside three human coders, building on previous research done in our group on the potential role of storytelling as an analytical framework in assessing design research projects. The findings indicated that storytelling frameworks could enhance research plans, but these were based on a single coder. This paper investigates the use of AI tools, specifically ChatGPT, as a cross-coder for analyzing storytelling frameworks in design research communication. It also assesses the reliability of two storytelling frameworks through multiple intercoder reliability evaluations. The results show that ChatGPT is promising as a coding tool in thematic analysis when given precise prompts, indicating potential for broader use in design research. Additionally, the storytelling framework with fewer codes exhibited higher intercoder reliability (ICR) compared to the more complex one, suggesting that a simplified framework might be more effective in capturing key narrative themes for general understanding.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Can an AI Agent Be Used for Intercoder Reliability in Thematic Analysis of Qualitative Data? An Experiment with Data on Storytelling

  • M. Shraddha Rao,
  • Vishal Singh

摘要

Thematic analysis of transcripts and qualitative data is frequently employed in design research and other fields like medicine. This analysis typically involves creating a coding scheme for a coder to classify data according to predefined codes. Using multiple coders mitigates individual biases, enhancing the reliability of the coding process and providing a measure of intercoder reliability, thus adding rigor to the research. In design research, intercoder reliability (ICR) is crucial in areas such as user experience research, qualitative data analysis, and content analysis of design artifacts. However, for extensive datasets, involving multiple coders can be resource-intensive. Hence, an AI Agent like GPT (Generative Pre-trained Transformers) could act as a coder, reducing the cost and effort of evaluating intercoder reliability. This study explores ChatGPT’s role as a coder alongside three human coders, building on previous research done in our group on the potential role of storytelling as an analytical framework in assessing design research projects. The findings indicated that storytelling frameworks could enhance research plans, but these were based on a single coder. This paper investigates the use of AI tools, specifically ChatGPT, as a cross-coder for analyzing storytelling frameworks in design research communication. It also assesses the reliability of two storytelling frameworks through multiple intercoder reliability evaluations. The results show that ChatGPT is promising as a coding tool in thematic analysis when given precise prompts, indicating potential for broader use in design research. Additionally, the storytelling framework with fewer codes exhibited higher intercoder reliability (ICR) compared to the more complex one, suggesting that a simplified framework might be more effective in capturing key narrative themes for general understanding.