Using Large Language Models to Automate Annotation and Part-of-Math Tagging of Math Equations

This paper explores the potential of leveraging Large Language Models (LLMs) for the tasks of automated annotation and Part-of-Math (POM) tagging of equations. Traditional methods for math term annotation and POM tagging rely heavily on manually crafted rules and limited datasets, which often result in scalability issues and insufficient adaptability to new domains. In contrast, LLMs, with their vast knowledge and advanced natural language understanding capabilities, present a promising alternative. Our methodology involves crafting prompts for LLMs to elicit answers that can be read as key-value pairs where the keys are math terms and the values are the corresponding annotations. We also investigate the effect on the performance of LLMs when we provide in the prompt different levels of context, such as the sentence or paragraph containing the input equation. The performance is evaluated by consistency between the ground truth and the output of LLMs. Consistency is assessed by a separate LLM session and with a different prompt. Our results show that when different levels of context are involved, the consistency rate of binary classification increased from 14.8% to 24.5%, and the favorable outcomes rate of multi-class classification increased from 47.1% to 77.5%. Finally, we conclude by discussing the implications of our findings for the future of mathematical knowledge management. We propose that LLMs could play a key role in automating the annotation and tagging of mathematical content, thereby enhancing the accessibility and utility of mathematical knowledge in digital libraries and beyond.