Scholarlink

<p>Despite the rapid advances in large language models (LLMs), understanding highly philosophical and culturally sophisticated writings remains underexplored. Ancient Indian works, in particular, pose unique challenges; they are rich in metaphor, moral argument, and contextual richness surpassing literal understanding or surface-level recall. In this experiment, we assessed if model type, retrieval strategy, and decoding parameters impact the accuracy and generalizability of LLMs when responding to multiple-choice questions (MCQs) from an ancient Indian philosophical work. We incorporated seven MCQs within a retrieval-augmented generation (RAG) setting and evaluated two leading LLMs ChatGPT-3.5 Turbo and Llama-2 in sixteen different settings varying in retrieval strategy (MPNet vs TF-IDF), temperature (1.0 vs 2.0), and top-p (0.1 vs 0.9). A total of 2,800 responses investigated two three-way ANOVAs to test the main and interaction effects of the variables. We find Llama-2 to systematically dominate ChatGPT-3.5 Turbo in both accuracy and generalizability. For decoding parameters, top-p emerged with a significant impact, with increasing values yielding improved performance, whereas temperature had a minor impact. Strikingly, no measurable difference was observed between the two retrieval strategies under the present experimental conditions. These findings suggest that, under the present experimental conditions, model architecture plays a more prominent role than retrieval strategy in driving performance. This study presents a new test framework for philosophical reasoning with LLMs and provides avenues for applying AI constructively in education, understanding of different cultures, and transmission of moral traditions.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Evaluating large language models for philosophical reasoning on ancient Indian texts

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年5月11日

查看全文
  • Sakshi Chauhan,
  • Isha Karn,
  • Varun Dutt

摘要

Despite the rapid advances in large language models (LLMs), understanding highly philosophical and culturally sophisticated writings remains underexplored. Ancient Indian works, in particular, pose unique challenges; they are rich in metaphor, moral argument, and contextual richness surpassing literal understanding or surface-level recall. In this experiment, we assessed if model type, retrieval strategy, and decoding parameters impact the accuracy and generalizability of LLMs when responding to multiple-choice questions (MCQs) from an ancient Indian philosophical work. We incorporated seven MCQs within a retrieval-augmented generation (RAG) setting and evaluated two leading LLMs ChatGPT-3.5 Turbo and Llama-2 in sixteen different settings varying in retrieval strategy (MPNet vs TF-IDF), temperature (1.0 vs 2.0), and top-p (0.1 vs 0.9). A total of 2,800 responses investigated two three-way ANOVAs to test the main and interaction effects of the variables. We find Llama-2 to systematically dominate ChatGPT-3.5 Turbo in both accuracy and generalizability. For decoding parameters, top-p emerged with a significant impact, with increasing values yielding improved performance, whereas temperature had a minor impact. Strikingly, no measurable difference was observed between the two retrieval strategies under the present experimental conditions. These findings suggest that, under the present experimental conditions, model architecture plays a more prominent role than retrieval strategy in driving performance. This study presents a new test framework for philosophical reasoning with LLMs and provides avenues for applying AI constructively in education, understanding of different cultures, and transmission of moral traditions.