Proof autoformalization aims to generate formal proofs from natural-language (NL) statements. While large language models (LLMs) show promise, they struggle with accuracy on complex proofs, and their size and cost limit broader use. We evaluate the LLM mistral-large-2411 on the Herald dataset, a large collection of aligned informal and formal Lean proofs. Our analysis reveals common model errors and structural issues in the dataset, including data leakage. Using iterative prompt refinement and practical fixes, we improve formalization quality. This work highlights both the potential and current limits of prompt-based autoformalization.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Exploring proof autoformalization with Mistral on Herald

  • Lucy Horowitz,
  • Michail Karatarakis,
  • Xuandi Ren,
  • Alejandro Sanchez Ocegueda

摘要

Proof autoformalization aims to generate formal proofs from natural-language (NL) statements. While large language models (LLMs) show promise, they struggle with accuracy on complex proofs, and their size and cost limit broader use. We evaluate the LLM mistral-large-2411 on the Herald dataset, a large collection of aligned informal and formal Lean proofs. Our analysis reveals common model errors and structural issues in the dataset, including data leakage. Using iterative prompt refinement and practical fixes, we improve formalization quality. This work highlights both the potential and current limits of prompt-based autoformalization.