Neural Style Transfer


We developed Neural Style Transfer, an algorithm based on deep learning and transfer learning that allows us to redraw a photograph in the style of any arbitrary painting with remarkable quality (Gatys, Ecker, Bethge, CVPR 2016, Gatys et al., CVPR 2017). In 2015, our arXiv preprint introducing the algorithm was the 9th most widely discussed academic paper world-wide in mainstream media, blogs and social media (see Altimetric 2015 Top 100 and article statistics).

Neural style transfer builds on our earlier work on texture synthesis (Gatys, Ecker, Bethge, NIPS 2015). The key insight of this line of research has been that deep feature spaces can be used as perceptual loss functions, which allow the separation and recombination of perceptually important image features in an unprecedented manner (Gatys, Ecker, Bethge, Current Opinion in Neurobiology 2017, Wallis et al., Journal of Vision 2017). Such perceptual losses are now used routinely in state-of-the-art methods for image processing tasks such as super resolution, inpainting, attribute manipulation, attribute-based image synthesis and sketch inversion.

style transfer example
Neural style transfer converts photographs into pieces of art.

Key Papers


L. A. Gatys, A. S. Ecker, and M. Bethge
Image Style Transfer Using Convolutional Neural Networks
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016
#texture transfer, #artistic style, #separating content from style, #convolutional neural networks
Code, URL, BibTex

L. A. Gatys, A. S. Ecker, M. Bethge, A. Hertzmann, and E. Shechtman
Controlling Perceptual Factors in Neural Style Transfer
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017
#texture transfer, #artistic style, #user control, #convolutional neural networks
Code, URL, Supplementary Material, BibTex

L. A. Gatys, A. S. Ecker, and M. Bethge
Texture and art with deep neural networks
Current Opinion in Neurobiology, 46, 178-186, 2017
#visual textures, #style transfer, #perceptual image synthesis, #cnns, #computational neuroscience
URL, PDF, BibTex

L. A. Gatys, A. S. Ecker, and M. Bethge
A Neural Algorithm of Artistic Style
arXiv, 2015
#artistic style, #convolutional neural networks, #separating content from style
URL, Details, BibTex

L. A. Gatys, A. S. Ecker, and M. Bethge
Texture Synthesis Using Convolutional Neural Networks
Advances in Neural Information Processing Systems 28, 2015
#texture synthesis, #ventral stream, #convolutional neural networks, #deep learning
Code, URL, PDF, Example textures, BibTex


University of Tuebingen BCCN CIN MPI