MCC: Multi-level Feature and Context-Aware Attention Mechanism with Consistent Distributions for Recipe Retrieval
摘要
With people’s increasing emphasis on healthy eating, food computing has become a significant research area, in which recipe retrieval is an essential part. In this paper, we are interested in retrieving food recipes from food images and vice versa. We present Multi-level Feature and Context-aware Attention Mechanism with Consistent Distributions for Recipe Retrieval (MCC). To reduce the distance between image modality and text modality, we introduce Maximum Mean Discrepancy and propose a novel triplet loss (TL-MMD), which outperforms traditional triplet loss by effectively aligning cross-modal distributions and enhancing convergence, thus achieving superior retrieval accuracy. Considering that a dish comprises multiple ingredients, with specific regions roughly corresponding to individual ingredients, we propose an encoder with multi-level features that innovatively integrates an advanced attention mechanism. This approach surpasses traditional CNN-based encoders by dynamically focusing on key image regions and fusing multi-resolution features, achieving richer and more detailed representations. Furthermore, we construct a Contextual Attention Module (CAM), targeting distinct regions in the image and individual words in the recipe simultaneously, to discover full latent alignments and infer region-word similarity with greater precision and interpretability than prior methods. Our model surpasses the competition by achieving state-of-the-art performance on Recipe1M, boasting an improvement of 2–4%. Through ablation experiments, we verify that each of our components contributes significantly to enhancing the performance, collectively establishing MCC as a superior solution for cross-modal recipe retrieval.