Despite the rapid advances in Visual Language Models (VLMs), these models struggle to recognize culture-specific food items. While VLMs are effective in recognizing popular cultural dishes, their performance is suboptimal for dishes that are unique but not widely known internationally. Specifically, VLMs often generate either generic labels or hallucinated names for dishes that are localized to a particular culture. As a result, retrieval-augmented generation (RAG), which retrieves relevant recipes as references for VLMs, emerges as a promising approach. Nevertheless, recipe retrieval, which is itself imperfect, could mislead VLMs into generating inaccurate or culturally inappropriate dish names. This paper presents a comparative study evaluating RAG-based food recognition against conventional approaches, including neural network-based recognition and standalone recipe retrieval. We propose an optimized hybrid framework that integrates the strengths of both VLMs and conventional techniques. The proposed framework achieves the best overall performance in recognizing multicultural dishes and demonstrates robustness in identifying out-of-distribution dishes from a different domain.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Food Recognition with Visual Language Models: Search Re-ranking or Retrieval-Augmented Generation?

  • Kian-Yu Gan,
  • Phuong-Anh Nguyen,
  • Chong-Wah Ngo

摘要

Despite the rapid advances in Visual Language Models (VLMs), these models struggle to recognize culture-specific food items. While VLMs are effective in recognizing popular cultural dishes, their performance is suboptimal for dishes that are unique but not widely known internationally. Specifically, VLMs often generate either generic labels or hallucinated names for dishes that are localized to a particular culture. As a result, retrieval-augmented generation (RAG), which retrieves relevant recipes as references for VLMs, emerges as a promising approach. Nevertheless, recipe retrieval, which is itself imperfect, could mislead VLMs into generating inaccurate or culturally inappropriate dish names. This paper presents a comparative study evaluating RAG-based food recognition against conventional approaches, including neural network-based recognition and standalone recipe retrieval. We propose an optimized hybrid framework that integrates the strengths of both VLMs and conventional techniques. The proposed framework achieves the best overall performance in recognizing multicultural dishes and demonstrates robustness in identifying out-of-distribution dishes from a different domain.