Theory-Informed vs. Example-Driven Prompting for LLM-Based Qualitative Data Coding in Educational Research
摘要
Large Language Models (LLMs) offer new possibilities for automating qualitative text coding, yet the optimal prompting strategy for coding academic literature remains unclear. This study compares two architectures on coding papers: a theory-informed zero-shot agent applying theoretical frameworks (i.e., Revised Bloom’s Taxonomy (RBT) and ICAP), and an example-driven few-shot agent that leverages in-context learning from manually coded papers. Using QWEN 3 and Gemini 2.5 Pro, we coded 13 academic papers on mathematics teaching software across 24 RBT sub-dimensions and 4 ICAP categories, evaluating agent performance against a manual coding baseline with Accuracy, F1-score, Cohen’s Kappa, and Matthews Correlation Coefficient. We hypothesised that the few-shot approach would outperform the zero-shot approach; however, results showed the opposite: the theory-only agent achieved more balanced and robust performance on the RBT framework, especially on imbalance-aware metrics. Category-level analysis revealed particular strength in conceptual and factual coding, while procedural and metacognitive dimensions remained more challenging. ICAP results were inconsistent across components, underscoring the construct-dependent nature of LLM performance. These findings highlight the value of well-structured theoretical prompts as a strong baseline for complex, multi-label qualitative coding, and emphasise the importance of evaluation metrics that address class imbalance when assessing LLM-assisted research workflows.