Scholarlink

<p>This study investigates the use of local Large Language Models (LLMs): Gemma, LLama, Mistral Small, and Phi, in assessing Spanish second language (L2) writing. While commercial models like ChatGPT have already been used for this task, they present privacy risks and reproducibility challenges. To evaluate local alternatives to ChatGPT, human raters were used to assess a third-semester assignment, evaluating 100 paragraphs to provide a baseline of comparison for the local LLMs’ assessment. Results showed varying performance: while some models approached the human baseline; others were inaccurate and inconsistent. On average, they performed better using a holistic rubric than an analytic one. A third experiment, averaging the top 3 models: Llama3.3:70b, Gemma2:9b, Mistral-Small:22b, achieved the highest agreement (<i>r</i> = 0.88), surpassing inter-human rating. A separate experiment testing RAG and fine-tuning techniques on the smallest model: Llama3.1:8b, showed no significant improvements with RAG but substantial gains in accuracy and consistency with fine tuning. Findings suggest that local LLMs show promise for Spanish L2 writing assessment. Local LLMs are preferable to proprietary ones because they protect privacy and reduce risks of data exposure, they may better support replicable workflows and may offer practical advantages for long term adoption.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Local language models for Spanish L2 writing assessment: human baselines, RAG, and fine-tuning

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年6月30日

查看全文
  • Giovanni Zimotti

摘要

This study investigates the use of local Large Language Models (LLMs): Gemma, LLama, Mistral Small, and Phi, in assessing Spanish second language (L2) writing. While commercial models like ChatGPT have already been used for this task, they present privacy risks and reproducibility challenges. To evaluate local alternatives to ChatGPT, human raters were used to assess a third-semester assignment, evaluating 100 paragraphs to provide a baseline of comparison for the local LLMs’ assessment. Results showed varying performance: while some models approached the human baseline; others were inaccurate and inconsistent. On average, they performed better using a holistic rubric than an analytic one. A third experiment, averaging the top 3 models: Llama3.3:70b, Gemma2:9b, Mistral-Small:22b, achieved the highest agreement (r = 0.88), surpassing inter-human rating. A separate experiment testing RAG and fine-tuning techniques on the smallest model: Llama3.1:8b, showed no significant improvements with RAG but substantial gains in accuracy and consistency with fine tuning. Findings suggest that local LLMs show promise for Spanish L2 writing assessment. Local LLMs are preferable to proprietary ones because they protect privacy and reduce risks of data exposure, they may better support replicable workflows and may offer practical advantages for long term adoption.