<p>Materials discovery and development are critical for addressing global challenges in renewable energy, sustainability, and advanced technology. Large language models (LLMs) offer unprecedented opportunities to accelerate materials research, yet their effective deployment requires domain-specific adaptation. Here we present large language models for materials (LLaMat), a family of foundational models for materials science, developed through continued pretraining of LLaMA models on 30 billion tokens derived from approximately 4 million materials science publications and crystallographic data. To develop a materials copilot, the models were adapted by instruction and task fine-tuning on 175,000 materials science question-answering pairs. Through evaluation across 42 tasks covering the entire spectrum of materials research, spanning natural language processing, structured information extraction and crystal generation, we demonstrate that LLaMat consistently outperforms state-of-the-art commercial LLMs (Claude, GPT and Gemini) while maintaining general linguistic capabilities. Beyond demonstrating the effectiveness of domain adaptation for practically deployable materials research copilots, our findings also reveal fundamental insights about LLM adaptation that may influence the development of specialized scientific artificial intelligence systems across domains. For instance, we identify increasing rigidity to domain adaptation in extensively pretrained LLMs such as LLaMA-3. This consistent pattern observed across the experiments suggests a previously unidentified ‘adaptation rigidity’, where overtrained LLMs exhibit increasing rigidity to domain adaptation.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A family of large language models for materials research with insights into model adaptability in continued pretraining

  • Dhruv Ahlawat,
  • Vaibhav Mishra,
  • Somaditya Singh,
  • Mohd Zaki,
  • Vaibhav Bihani,
  • Hargun Singh Grover,
  • Biswajit Mishra,
  • Santiago Miret,
  • Mausam,
  • N. M. Anoop Krishnan

摘要

Materials discovery and development are critical for addressing global challenges in renewable energy, sustainability, and advanced technology. Large language models (LLMs) offer unprecedented opportunities to accelerate materials research, yet their effective deployment requires domain-specific adaptation. Here we present large language models for materials (LLaMat), a family of foundational models for materials science, developed through continued pretraining of LLaMA models on 30 billion tokens derived from approximately 4 million materials science publications and crystallographic data. To develop a materials copilot, the models were adapted by instruction and task fine-tuning on 175,000 materials science question-answering pairs. Through evaluation across 42 tasks covering the entire spectrum of materials research, spanning natural language processing, structured information extraction and crystal generation, we demonstrate that LLaMat consistently outperforms state-of-the-art commercial LLMs (Claude, GPT and Gemini) while maintaining general linguistic capabilities. Beyond demonstrating the effectiveness of domain adaptation for practically deployable materials research copilots, our findings also reveal fundamental insights about LLM adaptation that may influence the development of specialized scientific artificial intelligence systems across domains. For instance, we identify increasing rigidity to domain adaptation in extensively pretrained LLMs such as LLaMA-3. This consistent pattern observed across the experiments suggests a previously unidentified ‘adaptation rigidity’, where overtrained LLMs exhibit increasing rigidity to domain adaptation.