Scholarlink

Large Language Models (LLMs) have demonstrated strong capabilities in NLP tasks through extensive pretraining. However, this paper highlights their limited ability to perform language-unaware sandhi-splitting (splitting the conjoined words as per grammar rules) in Sanskrit—a task requiring metacognitive reasoning. We introduce a dataset of a hundred Sanskrit words with their splits and evaluate Sanskrit-unaware LLMs and human annotators on this task. To make it a metacognitive task, with each sample we provide two reference examples closest to the sample (yet challenging since at least one of the splits in each reference is different from the sample’s split), which are helpful in making the correct guess. Our findings reveal a significant gap in humans’ and LLMs’ performance, with a Word Error Rate (WER) difference of up to 85.45%, indicating the absence of metacognitive skills in current LLMs.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Assessing Metacognitive Ability of LLMs Using a Language-Unaware Sandhi-Splitting Paradigm

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Rajat Verma,
  • Nandan Paralikar,
  • Manikandan Ravikiran,
  • Rohit Saluja,
  • Ganesh Ramakrishnan

摘要

Large Language Models (LLMs) have demonstrated strong capabilities in NLP tasks through extensive pretraining. However, this paper highlights their limited ability to perform language-unaware sandhi-splitting (splitting the conjoined words as per grammar rules) in Sanskrit—a task requiring metacognitive reasoning. We introduce a dataset of a hundred Sanskrit words with their splits and evaluate Sanskrit-unaware LLMs and human annotators on this task. To make it a metacognitive task, with each sample we provide two reference examples closest to the sample (yet challenging since at least one of the splits in each reference is different from the sample’s split), which are helpful in making the correct guess. Our findings reveal a significant gap in humans’ and LLMs’ performance, with a Word Error Rate (WER) difference of up to 85.45%, indicating the absence of metacognitive skills in current LLMs.