<p>The study delves into the implications of Artificial Intelligence (AI), Large Language Model (LLM) adoption on qualitative research, particularly thematic synthesis, by using content analysis from multifaceted perspectives, i.e., the author’s qualitative assessments and SWOT analysis. Utilising a systematic literature search, the author examines 65 AI-generated themes identified by the AI-LLM tool'ChatPDF' from 17 pieces of literature, focusing on context accuracy, textual patterns, review depth, inclusivity of sensitive topics, word counts, and SWOT analysis. Findings show a 56.92% context matching, which indicates a deeper and relevant insight into the AI-generated themes, thereby fostering thematic progression, and 89.23% non-repetitiveness in textual patterns, pointing to non-repetitive nature of the texts within the theme descriptions. Review depths vary, indicating diverse levels of In-depth, average, and surface-level review. 17 of the 65 AI-generated themes (26.15%) did not include sensitive topics, with only a handful of 4.61% addressing sensitive topics, which may generate biased themes. On the other hand, the SWOT analysis highlights ChatPDF's strengths in speedy interpretation, text summarisation, decent contextual accuracy, non-repetitive textual patterns, trend or gaps identification, translation feature, average word counts, improved research structure and opportunities like reducing researcher burnout, ease of adoption, conversational ability, connect concepts, enhanced methodologies, equitable access, and enhancing collaboration with the possibility to improve researcher’s experience. However, it needs to improve contextual understanding, minor text repetitions, sensitive topic inclusion, statistical data extraction, potential algorithmic biases, privacy concerns, conversion of non-OCR PDF files, and transparency in its trained datasets. AI-LLMs offer improvements in qualitative research, specifically in thematic progression. Researchers can leverage LLMs for diverse theme elaboration and summarisation through multiple prompts. At the same time, AI developers must enhance their systems' contextual accuracy by flagging errors or bias, adapting to varied study types, and improving statistical data extraction. Higher educational institutions and publishers should have strong policies to validate AI-generated content and provide training for ethical adoption while protecting users' privacy. This study contributes significantly to researchers who intend to use LLM in qualitative research. It addresses the impact of LLM across all stakeholders, i.e., researchers, AI developers, educational institutions and publishers, while emphasising its ethical, transparent, and safe adoption. This study further advances the discussion on the ever-changing roles of AI-human collaboration in academic research.</p>

A qualitative assessment of the accuracy of AI-LLM in academic research

期刊论文
发表日期：

2025年4月28日

查看全文

Chanlang Ki Bareh

摘要

The study delves into the implications of Artificial Intelligence (AI), Large Language Model (LLM) adoption on qualitative research, particularly thematic synthesis, by using content analysis from multifaceted perspectives, i.e., the author’s qualitative assessments and SWOT analysis. Utilising a systematic literature search, the author examines 65 AI-generated themes identified by the AI-LLM tool'ChatPDF' from 17 pieces of literature, focusing on context accuracy, textual patterns, review depth, inclusivity of sensitive topics, word counts, and SWOT analysis. Findings show a 56.92% context matching, which indicates a deeper and relevant insight into the AI-generated themes, thereby fostering thematic progression, and 89.23% non-repetitiveness in textual patterns, pointing to non-repetitive nature of the texts within the theme descriptions. Review depths vary, indicating diverse levels of In-depth, average, and surface-level review. 17 of the 65 AI-generated themes (26.15%) did not include sensitive topics, with only a handful of 4.61% addressing sensitive topics, which may generate biased themes. On the other hand, the SWOT analysis highlights ChatPDF's strengths in speedy interpretation, text summarisation, decent contextual accuracy, non-repetitive textual patterns, trend or gaps identification, translation feature, average word counts, improved research structure and opportunities like reducing researcher burnout, ease of adoption, conversational ability, connect concepts, enhanced methodologies, equitable access, and enhancing collaboration with the possibility to improve researcher’s experience. However, it needs to improve contextual understanding, minor text repetitions, sensitive topic inclusion, statistical data extraction, potential algorithmic biases, privacy concerns, conversion of non-OCR PDF files, and transparency in its trained datasets. AI-LLMs offer improvements in qualitative research, specifically in thematic progression. Researchers can leverage LLMs for diverse theme elaboration and summarisation through multiple prompts. At the same time, AI developers must enhance their systems' contextual accuracy by flagging errors or bias, adapting to varied study types, and improving statistical data extraction. Higher educational institutions and publishers should have strong policies to validate AI-generated content and provide training for ethical adoption while protecting users' privacy. This study contributes significantly to researchers who intend to use LLM in qualitative research. It addresses the impact of LLM across all stakeholders, i.e., researchers, AI developers, educational institutions and publishers, while emphasising its ethical, transparent, and safe adoption. This study further advances the discussion on the ever-changing roles of AI-human collaboration in academic research.