Deep Network-Based Computational Transfer of Artistic Style in Art Analysis

Some of the most successful computational approaches to the analysis of natural photographs and videos are based on deep neural networks trained with large corpora of representative photographs. Unfortunately, such trained networks have but modest performance on analogous tasks when they are applied directly to fine art paintings and drawings, including semantic segmentation, captioning, and question answering. The obvious program would be to train (or transfer train) existing network architectures with large corpora of representative fine art images but even the largest art corpora are far too small to yield high accuracy on fine art images. We describe an alternate approach in which separate deep networks computationally transfer the style of representative artworks and movements onto large corpora of natural photographs to thereby create surrogate artworks. Such surrogate artworks might include a helicopter rendered in the style of Claude Monet or a portrait of Oprah Winfree rendered in the style of Caravaggio. Separate application-specific deep networks trained with such surrogate artworks show state-of-the-art performance on image segmentation. The accuracy of these networks implies that the early, low-level feature extractors are most altered to apply to fine art. Such creation of surrogate art images should find wide use throughout computational studies of fine art paintings and drawings.