Introduction

It's a course project I advised. Two brilliant and passionated master students at NYU, Megan Hardy and Sumanto Pal, start their adventure of image processing and deep learning. Different objects of a painting typically have distinctively different styles i.e. textures and color compositions while previous neural painting works abstract a global style from a painting and apply it globally onto a photo. In this project, we separate objects of interest in both painting and real content photo and apply an art style only to the corresponding objects. For example, abstract style from a flower in a painting and apply it onto a flower in a real content photo. It creates unique aesthetic experiences.

Object segmentation

As the first challenge, we use Grabcut to segment objects of interest. Grabcut is a segmentation technique which requires minimal level of human intervene while is able to achieve highly details-preserving results.

Fig. 1 Grabcut example. One can simply place a bounding box around the object of interest and may need to roughly indicate object with white marker and non-object with black marker. Then the object can be segmented nicely.

Painting with multiple styles

After some intermediary and post processing, we are able to abstract styles from objects in paintings and apply them onto corresponding objects in real content photos.

Fig. 2 Flower style on flower.
Fig. 3 Jordon in the space (background styled only).
Fig. 3 Separated style for foreground and background.

Publication

Hardy, Megan, and Sumanto Pal, "Split Consideration for Foreground and Background Painting Using Artificial Neural Networks," Proceedings of the 2017 ACM on Multimedia Conference, 2017.

References

Back to Yuan's homepage