Scholarlink

<p>Based on existing learning models, multimodal approaches have demonstrated promising performance in the realm of few-shot learning, owing to contrastive language-image pretraining. However, shortcomings persist in multimodal fusion methods, particularly in aligning textual and visual features across different granularity levels, and in the independent feature extraction by encoders lacking interaction. Therefore, this paper proposes MFAFD: a cascaded model for few-shot learning featuring parameter free attention mechanisms and a finite discrete space. Initially, the model employs a parameter free attention module in the pretraining phase to facilitate cross-modal interactions, enhancing alignment between spatial features of images and generated text prior to extracting global features from images via CLIP. This bidirectional update of textual and visual information addresses the issue of feature alignment. During training, the model leverages a representation based on Finite Discrete Space (FDS), constructing a finite discrete space foundation for textual and image features, effectively bridging modal differences. Ultimately, using text as a baseline, the model predicts image classification based on similarity weights between images and text. Through quantitative and qualitative analyses, this study demonstrates that parameter free attention mechanisms and finite discrete space modules significantly enhance the performance of cascaded multimodal aggregation models. The model exhibits robust performance in few-shot classification across multiple datasets. The code is available at <a href="https://github.com/turelove999/MFDFA-dj">https://github.com/turelove999/MFDFA-dj</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

MFAFD: a few-shot learning method for cascading models with parameter free attention and finite discrete space

  • 期刊论文

  • 发表日期：

    2025年3月6日

查看全文
  • Lixia Xue,
  • Jiang Dong,
  • Ronggui Wang,
  • Juan Yang

摘要

Based on existing learning models, multimodal approaches have demonstrated promising performance in the realm of few-shot learning, owing to contrastive language-image pretraining. However, shortcomings persist in multimodal fusion methods, particularly in aligning textual and visual features across different granularity levels, and in the independent feature extraction by encoders lacking interaction. Therefore, this paper proposes MFAFD: a cascaded model for few-shot learning featuring parameter free attention mechanisms and a finite discrete space. Initially, the model employs a parameter free attention module in the pretraining phase to facilitate cross-modal interactions, enhancing alignment between spatial features of images and generated text prior to extracting global features from images via CLIP. This bidirectional update of textual and visual information addresses the issue of feature alignment. During training, the model leverages a representation based on Finite Discrete Space (FDS), constructing a finite discrete space foundation for textual and image features, effectively bridging modal differences. Ultimately, using text as a baseline, the model predicts image classification based on similarity weights between images and text. Through quantitative and qualitative analyses, this study demonstrates that parameter free attention mechanisms and finite discrete space modules significantly enhance the performance of cascaded multimodal aggregation models. The model exhibits robust performance in few-shot classification across multiple datasets. The code is available at https://github.com/turelove999/MFDFA-dj.