Scholarlink

<p>Vision-language models can be used for open-vocabulary object detection. The existing methods suffer from low matching accuracy between prompt and image regions, as well as limited generalization capability as they adopt a data-centralized model training approach that ignores data heterogeneity. To alleviate these issues, we propose a federated fine-grained prompts learning method called FFPLearning, for open-vocabulary object detection using vision-language models. Specifically, FFPLearning quantifies the quality of proposals using pre-fused EoG (Energy of Gradient) and IoU (Intersection over Union) scores and organizes them into individual groups. Then learnable fine-grained prompts are trained to align the grouped region proposals in the feature space. A momentum update algorithm is designed to assess the quality of each participating client in the federated learning. Additionally, a Transformer-based feedback aggregation algorithm is designed to thoroughly leverage the semantic information from prompts and aggregate them based on the qualities of clients. Comprehensive evaluations on COCO and LVIS datasets demonstrate that FFPLearning is very effective, with +5.8 Novel AP50 and +3.3 APr improvements compared with existing state-of-the-art methods.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Federated fine-grained prompts for vision-language models based on open-vocabulary object detection

  • 期刊论文

  • 发表日期：

    2025年4月4日

查看全文
  • Yu Li

摘要

Vision-language models can be used for open-vocabulary object detection. The existing methods suffer from low matching accuracy between prompt and image regions, as well as limited generalization capability as they adopt a data-centralized model training approach that ignores data heterogeneity. To alleviate these issues, we propose a federated fine-grained prompts learning method called FFPLearning, for open-vocabulary object detection using vision-language models. Specifically, FFPLearning quantifies the quality of proposals using pre-fused EoG (Energy of Gradient) and IoU (Intersection over Union) scores and organizes them into individual groups. Then learnable fine-grained prompts are trained to align the grouped region proposals in the feature space. A momentum update algorithm is designed to assess the quality of each participating client in the federated learning. Additionally, a Transformer-based feedback aggregation algorithm is designed to thoroughly leverage the semantic information from prompts and aggregate them based on the qualities of clients. Comprehensive evaluations on COCO and LVIS datasets demonstrate that FFPLearning is very effective, with +5.8 Novel AP50 and +3.3 APr improvements compared with existing state-of-the-art methods.