<p>Few-Shot Object Detection (FSOD) aims to detect and localize objects from novel categories using only a limited number of annotated examples. With the rapid advancement of large-scale foundation models, FSOD has made remarkable progress by leveraging their extensive cross-modal prior knowledge. However, existing foundation modelbased FSOD methods still face two major challenges. First, most methods rely solely on unimodal embeddings for category representation, failing to fully exploit the complementary information from both visual and textual modalities, which limits the discriminability of category features. Second, a semantic gap exists between different modalities’ features extracted from pre-trained foundation models, making effective alignment difficult under few-shot constraints. To address the above issues, we propose a Multimodal Enhancement and Alignment Framework built upon Large-scale Foundation Models for few-shot object detection. The framework comprises a Multimodal Prototype Enhancement (MPE) module that constructs more discriminative category prototype representations by effectively fusing visual features with textual embeddings. And a Foundation Model Aligner (FMA) that aligns multimodal category features in the semantic space of large language models through a lightweight adaptive network and instruction prompts, without requiring fine-tuning of the foundation models. The FMA enhances the consistency of category representations and enables efficient cross-modal semantic alignment. Extensive experiments on multiple FSOD benchmarks demonstrate that our approach effectively improves category prototype discriminability and semantic alignment between foundation models, achieving state-of-the-art performance and stronger generalization under few-shot settings.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Large-scale foundation models based multimodal enhancement and alignment framework for few-shot object detection

  • Duorui Wang,
  • Zhiwan Fang,
  • Xiaowei Zhao

摘要

Few-Shot Object Detection (FSOD) aims to detect and localize objects from novel categories using only a limited number of annotated examples. With the rapid advancement of large-scale foundation models, FSOD has made remarkable progress by leveraging their extensive cross-modal prior knowledge. However, existing foundation modelbased FSOD methods still face two major challenges. First, most methods rely solely on unimodal embeddings for category representation, failing to fully exploit the complementary information from both visual and textual modalities, which limits the discriminability of category features. Second, a semantic gap exists between different modalities’ features extracted from pre-trained foundation models, making effective alignment difficult under few-shot constraints. To address the above issues, we propose a Multimodal Enhancement and Alignment Framework built upon Large-scale Foundation Models for few-shot object detection. The framework comprises a Multimodal Prototype Enhancement (MPE) module that constructs more discriminative category prototype representations by effectively fusing visual features with textual embeddings. And a Foundation Model Aligner (FMA) that aligns multimodal category features in the semantic space of large language models through a lightweight adaptive network and instruction prompts, without requiring fine-tuning of the foundation models. The FMA enhances the consistency of category representations and enables efficient cross-modal semantic alignment. Extensive experiments on multiple FSOD benchmarks demonstrate that our approach effectively improves category prototype discriminability and semantic alignment between foundation models, achieving state-of-the-art performance and stronger generalization under few-shot settings.