EAGER: An Exploratory Analysis of LLMs in GuEssing Bilingual WoRd Translation
摘要
Large Language Models (LLMs) are artificial intelligence systems that are designed to understand and generate human language. LLMs are widely in use for tasks like text generation, translation, summarization, code writing, communication improvement, and automation. LLMs trained in one language enhance natural language processing tasks in that language but lack the ability to guess, unlike humans, the meaning or synonyms in other related languages. In this paper, we propose a dataset of 160 Hindi-Sanskrit word pairs and perform their analysis on multiple LLMs, unaware of the Sanskrit language. Similarly, we use the 160 Sanskrit-Hindi word pairs to perform analysis on multiple LLMs unaware of the Hindi language. The dataset and analysis demonstrate LLMs’ limited capacity for metacognition, i.e., cognitive speculation and inference in language translation tasks. To validate our hypothesis, we perform inference from multiple LLMs on our dataset and observe that they have 50.40% less accuracy compared to humans in guessing the meaning of the Hindi word in the Sanskrit language. Similarly, we observe that LLMs achieve 53.28% less accuracy compared to humans in guessing the meaning of the Sanskrit word in the Hindi language.