Tattoo recognition and retrieval remain challenging tasks due to the diverse and intricate nature of tattoo designs. Existing approaches typically rely on visual features extracted from convolutional neural networks (CNNs), which may fail to capture the rich semantic information embedded in tattoos. In this paper, we propose a novel framework that integrates state-of-the-art visual features with textual descriptions generated by a multimodal large language model (MLLM). We explore the impact of different prompts on the quality of the generated captions and use CLIP to create textual embeddings. By combining cosine similarity scores from both modalities, our approach achieves superior performance in tattoo retrieval tasks. Experimental results demonstrate that our method outperforms traditional visual-only approaches, highlighting the importance of leveraging multimodal data for tattoo recognition. To the best of our knowledge, this is the first work to combine MLLM-generated textual descriptions with visual features for tattoo retrieval.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Multimodal Tattoo Recognition by Combining Visual Features and LLM-Generated Captions

  • Annette Morales-González,
  • Heydi Méndez-Vázquez,
  • Milton García-Borroto

摘要

Tattoo recognition and retrieval remain challenging tasks due to the diverse and intricate nature of tattoo designs. Existing approaches typically rely on visual features extracted from convolutional neural networks (CNNs), which may fail to capture the rich semantic information embedded in tattoos. In this paper, we propose a novel framework that integrates state-of-the-art visual features with textual descriptions generated by a multimodal large language model (MLLM). We explore the impact of different prompts on the quality of the generated captions and use CLIP to create textual embeddings. By combining cosine similarity scores from both modalities, our approach achieves superior performance in tattoo retrieval tasks. Experimental results demonstrate that our method outperforms traditional visual-only approaches, highlighting the importance of leveraging multimodal data for tattoo recognition. To the best of our knowledge, this is the first work to combine MLLM-generated textual descriptions with visual features for tattoo retrieval.