<p>In this study, we investigate how and why a range of large language models (LLMs) that we have previously worked with (Claude, Gemma, MiniMax, chatGPT, and chatGPT-3 model fine-tuned locally) perform automated assessments for foreign language learning and teaching. Our focus is on whether and how AI-based assessment replicates or augments the role of human grading in the assessment of German as a Foreign Language (DaF). This study used a three-pronged approach for evaluation. First, it uses general-purpose language models to grade answers based on corrected sheets. Then, it is graded using a Gemma 3 model that has been specially trained on exam data. Finally, human graders weigh in using both a grading rubric and a visual approach, where correct answers are laid over student responses to check for accuracy. The researchers measured the efficacy of these methods by examining their agreement using statistics, such as Cohen’s kappa and root mean square error (RMSE), to gauge accuracy, reliability, and practicality. The results show that LLMs for general use are unreliable and well behind humans, with performance from trivial to low for all evaluators combined. The fine-tuned Gemma 3 model showed low/mid-range improvement but still fell well below the human evaluators. Among the AIs used in this study, Claude Opus 4 was the best-performing AI and was reliably low to mid-range in reliability to human evaluators for both assessment options. Although the overlay method was not as rapid, it was the most accurate and consistent. Human grading remains the standard for reliability in language assessment, while current AI systems are potentially only useful as augmenting tools, as opposed to replacements. Fine-tuned local models show the possibility of domain-specific adjustment, but they must be improved from their current state to be useful for high-stakes automated grading.</p>