Multi-modal Entity Linking Model Based on Knowledge Distillation
摘要
Multi-modal Entity Linking (MEL) integrates Natural Language Processing (NLP) and Computer Vision to link entity mentions in text and images to knowledge bases. Recent advances in Large Language Models (LLMs) have significantly enhanced MEL performance by leveraging multi-modal data and complex reasoning. However, the large scale and complexity of these models present challenges in terms of efficiency and resource consumption. This paper introduces a multi-teacher hybrid distillation framework that combines both multi-modal LLMs (MLLMs) and traditional LLMs as teacher models to guide the fine-tuning of a lightweight student model. We address key challenges such as the need for diverse teacher models, task-specific data generation for distillation, and maintaining balanced capabilities in the student model. MEL is decomposed into three core stages: mention semantic enhancement, entity semantic comparison, and entity linking prediction. We identify three key capabilities necessary for MEL: multi-modal semantic understanding, context reasoning, and structured output. Experimental results show that our student model achieves performance comparable to larger models on benchmark datasets (Wiki-MEL, Richpedia-MEL, and Wikidiverse), while improving inference speed by a factor of two and reducing resource consumption by over 80%. This approach provides an efficient solution for MEL, offering a strong balance between performance and computational efficiency.