Testing Procedure Generation Based on Testing Requirements for Automotive Components with LLM Fine-Tuning
摘要
While a well-trained Transformer-based Large Language model has been proven to own an excellent ability to answer any question based on the input, the high accuracy of the answer output, especially for testing procedure generation based on testing requirements for automotive components focused in this paper, usually relies on either pre-training data with domain knowledge about the testing or input prompts with clear requirement instruction by well-dedicated prompt engineering. Since expensive pre-training data or instructing an overwhelming volume of untraceable tacit expertise in the input prompts is required in those general LLMs for procedure generation, it is not easy for most industries to adopt LLMs for their practical applications. This paper explores the model performance for the case when only limited domain-specific training data, around two hundred records, is available for our chosen models with fine-tuning and further provides insight into what level we can rely on fine-tuning for testing procedure generation based on Testing requirements for automotive components with BLEU evaluation. We chose Flan-T5-large and Mistral-7B as our target models with consideration for the model sizes with affordable fine-tuning methods and their few-shot prediction performance. Our results show that both fine-tuned target models can generate our expected procedures with more than 0.8 BLEU on average, while Mistral-7B can even derive new content for the procedure.