Exploiting Diffusion Model as Prompt Generator for Object Localization

Recently, diffusion models have shown unprecedented power in text-to-image generation. The intermediate features in well-trained text-to-image diffusion models have been proven to contain basic semantic and layout information of the synthesized image. Based on such findings, we present a Diffusion-model-based Prompt generator for Object Localization, named as DPOL. By providing proper text guidance to DPOL, the corresponding object in an image can be localized within two steps: (1) Prompt generation. Conditioned on the text guidance, the image is first inverted into its corresponding latent code and then reconstructed by the diffusion model. The attention maps produced by the diffusion model are used as the location prompt, which contain the coarse position information of the interested objects; (2) Location refinement. The Segment Anything Model (i.e., SAM) is used to get a more accurate position based on the location prompt, which is transformed into the format (in detail, box) that is compatible with SAM. Extensive experiments are conducted to show that our DPOL achieves comparable performance with existing open-vocabulary localization methods, even DPOL requires neither training nor fine-tuning.