The test data often exhibit the characteristics of multimodality and environmental complexity, and these characteristics have brought significant challenges to the detection and analysis work. The existing methods have deficiencies in interacting with the system through natural language processing, resulting in additional human resource consumption. The frontier topic of the integrated application of the You Only Look Once World (YOLO-World) and the large language model (LLM) is focused on by this paper. Firstly, the pre-trained LLM is used to tokenize the natural language and understand the user task, and then the tokenized data is passed into the agent designed based on YOLO-World to detect the entities of the test task. Then, based on prompt engineering, YOLO-World and LLM are innovatively integrated and the EdLLM is proposed, which creatively integrates the multimodal data, aiming to break the barrier between vision and language and realize the collaborative interaction between visual information and semantic information. Finally, it is applied in the field of test data to provide a more efficient, accurate, and intelligent solution. Compared with the existing open source model, this model can obtain entity information more effectively and quickly, and can detect entities based on prior knowledge combined with LLM under the condition of limited computing resources, which opens up a new way for the further development of related fields.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

EdLLM: A Novel Entity Detection Method for Test Data Integrating YOLO-World and LLM

  • Jintao Liu,
  • Chen Feng,
  • Guang Jin,
  • Jun Fan

摘要

The test data often exhibit the characteristics of multimodality and environmental complexity, and these characteristics have brought significant challenges to the detection and analysis work. The existing methods have deficiencies in interacting with the system through natural language processing, resulting in additional human resource consumption. The frontier topic of the integrated application of the You Only Look Once World (YOLO-World) and the large language model (LLM) is focused on by this paper. Firstly, the pre-trained LLM is used to tokenize the natural language and understand the user task, and then the tokenized data is passed into the agent designed based on YOLO-World to detect the entities of the test task. Then, based on prompt engineering, YOLO-World and LLM are innovatively integrated and the EdLLM is proposed, which creatively integrates the multimodal data, aiming to break the barrier between vision and language and realize the collaborative interaction between visual information and semantic information. Finally, it is applied in the field of test data to provide a more efficient, accurate, and intelligent solution. Compared with the existing open source model, this model can obtain entity information more effectively and quickly, and can detect entities based on prior knowledge combined with LLM under the condition of limited computing resources, which opens up a new way for the further development of related fields.