Capabilities and limitations of AI Large Language Models (LLMs) for materials criticality research

Generative AI chatbots such as ChatGPT are built upon large language models (LLMs). These tools have attracted significant attention due to their ability to generate narrative text, and their future potential to compile general search results and assist in data analysis and visualization. While the number and range of use cases for which these tools have demonstrated capabilities continues to grow rapidly, concerns have arisen about the accuracy and completeness when used in a research context. For example, in early 2023 there were reports of “hallucinations” where the chatbot returned fictitious results and references. This Brief Report considers the capabilities and limitations of LLMs for use in materials criticality research. Exploratory experiments were performed with several chatbots based on different underlying LLMs to examine the quality and extent of data coverage (e.g., accuracy, resolution, timeliness), the ability to recognize uncertainty and reconcile divergent data, and new directions that might be uniquely enabled by this technology.