CoverGAN: cover photo generation from text story using layout guided GAN
摘要
Generating cover photos from story text is a non trivial challenge to solve. Existing approaches focus on generating only images from given text prompt. To the best of our knowledge, non of these approaches focus on generating cover photos from a text story. The paper addresses this issue by introducing multi-object image generation with text title from a text story. We split the problem into three steps:- understanding semantics of text story, predicting layout of objects, and generating a cover photo. At start, a semantic relation was encoded between text story objects using a Scene graph, then features from graph neural network were concatenated with single object features from scene graph to create an object layout. All of these features were then passed on to the image generating part. Image generation was further divided into two phases. In the first phase, the image is generated using a scene graph image generation model. While in the second phase, the results of first phase were further enhanced using image translation model conditioned on the object layout. In final phase we generated title of the given story based on generated image. In our experiments, we used custom dataset of text stories with three animal categories along with the COCO dataset. For the image generating part, we evaluated our approach with state of the art models known as scene_gen and sg2im. Our method generated high-resolution informative cover photo with story title by positioning the objects at right locations as specified in the text story.