Benchmarking Modern Large Language Models on MSC Nastran BDF Input Files: A Comprehensive Study

The integration of Large Language Models (LLMs) into Computer Aided Engineering (CAE) workflows presents a novel opportunity to enhance pre-processing efficiency and accuracy. This study explores the application of open-source Large Language Models (LLMs) in interpreting and manipulating MSC Nastran Bulk Data Format (BDF) files, a critical solver-input format widely used in Finite element analysis (FEA) for industries such as aerospace and automotive engineering. Despite advancements in pre-processing CAE tools, modifying BDF files still demands extensive manual intervention and specialized expertise. To address these challenges, this research evaluates the performance of various LLMs, including chatbot and reasoning models in handling tasks such as simple queries, data structurization and cross-reference queries, and parameter modifications. The study benchmarks models like DeepSeek-V3, DeepSeek-R1 and Qwen2.5’s variants against traditional tools like Pynastran, using exact-match as metrics. Results reveal that chatbot models perform adequately compared with reasoning models, and moreover, models with small sizes demonstrate a relatively satisfactory performance in some certain tasks. This work underlines the potential of LLMs to streamline CAE pre-processing, while also identifying pathways for future enhancements.