Instruction Tuning Large Language Models for Multimodal Relation Extraction Using LoRA

The rapid proliferation of multimodal data on social platforms, particularly text-image pairs, has necessitated advanced methodologies for Multimodal Relation Extraction (MRE) to accurately identify semantic relations between annotated entities. Traditional approaches to MRE often suffer from limited generalizability across different datasets. To address the challenge, we propose to Instruction-Tune Large Language Models for MRE (ITMRE), which further utilizes Low-Rank Adaptation (LoRA) to tailor the Multimodal Large Language Models (MLLMs) for MRE. Our approach simplifies the extraction process using a two-stage multiple-choice question-answer template to first identify the types of two annotated entities in the text and then infer their relations. This structured methodology, combined with targeted LoRA tuning, allows for efficient model optimization without extensive retraining. We demonstrate the effectiveness of ITMRE through comprehensive experiments on the MNRE dataset, where it significantly outperforms existing methods and leading-edge language models such as GPT-3.5 and GPT-4 in terms of precision, recall, and F1 score, with an improvement in F1 scores by over 8%.