Fine-Tuning Strategies for Named Entity Recognition in Medical Text Images Using VLMs
摘要
Named Entity Recognition (NER) in biomedical research articles is an essential task for the automation of data extraction and enabling downstream clinical analysis. In this work, a pre-existing Vision-Language Model (Llama 11B-Vision) on the annotated dataset is used to improve its ability to recognize and extract named entities from medical research paper images accurately. A high-capacity model, Llama 90B-Vision is used to generate NER annotations that serve as ground truth value. The baseline performance of the smaller model, Llama 11B-Vision is compared to this ground truth values using typical precision, recall, and F1 score metrics for its NER predictions. This smaller model is next fine-tuned on a custom dataset developed with ideal annotations to enhance on its domain-specific performance. This paper focuses on data preprocessing, model testing, and driver integration for automated NER extraction. Experimental results show substantial improvements with respect to precision after fine-tuning, substantiating the effectiveness of the suggested approach to NER for medical research papers. Results obtained shows precision of 80% for the fine-tuned Llama 11B model over Llama 11B model without fine-tuning, which achieves a precision of only 72%.