Monet

Applying Style Transfer to Portraits

in ,

Hunter and I have been focusing on reverse engineering the three most famous paintings according to Google as well as a hand selected piece from the National Gallery.  These art works are The Mona Lisa, The Starry Night, The Scream, and Woman With A Parasol.

We also just recently got Style Transfer working on our own Tensor Flow system. So naturally we decided to take a moment to see how a neural net would paint using the four paintings we selected plus a second work by Van Gogh, his Self-Portrait (1889).  

Below is a grid of the results.  Across the top are the images from which style was transferred, and down the side are the images the styles were applied to. (Once again a special thanks to deepdreamgenerator.com for letting us borrow some of their processing power to get all these done.)

It is interesting to see where the algorithm did well and where it did little more than transfer the color and texture.  A good example of where it did well can be seen in the last column. Notice how the composition of the source style and the portrait it is being applied to line up almost perfectly. Well as could be expected, this resulted in a good transfer of style.

As far as failure. it is easy to notice lots of limitations. Foremost, I noticed that the photo being transferred needs to be high quality for the transfer to work well. Another problem is that the algorithm has no idea what it is doing with regards to composition.  For example, in The Scream style transfers, it paints a sunset across just about everyone's forehead.

We are still in processing of creating a step by step animation that will show one of the portraits having the style applied to it.  It will be a little while thought cause I am running it on a computer that can only generate one frame every 30 minutes.  This is super processor intensive stuff.

While processor is working on that we are going to go and see if we can't find a way to improve upon this algorithm.

 

 

 

 

Capturing Monet's Style with a Robot

in ,

As we gather data in an attempt to recreate the style and brushstroke of old masters with Deep Learning, we thought we would show you one of the ways we are collecting data.  And it is pretty simple actually. We are hand painting brushstrokes with a 7BOT robotic arm and recording the kinematics behind the strokes.  It is a simple trace and record operation where the robotic arms position is recorded 30 times a second and saved to a file.

As can be seen in the picture above, all Hunter had to do was trace the brush strokes he saw in the painting.  He did this for a number of colors and when he was done, we were able to play the strokes back to see how well the robot understood our instructions.  As can be seen in the following video, the playback was a disaster.  But that doesn't matter to us that much.  We are not interested in the particular strokes as much as we are in analyzing them for use in the Deep Learning algorithm we are working on.

Woman With A Parasol is the fourth Masterpiece we have begun collecting data for.  As this is an open source project, we will be making all the data we collect public.  For example, if you have a 7Bot, or similar robotic arm with 7 actuators, here are the files that we used to record the strokes and make the horrible reproduction.

 

 

Selecting Masterpieces to Recreate with Our Robot

in ,

Spent afternoon with Hunter exploring the National Gallery of Art to decide on the next masterpiece we are going to recreate and analyze with our robot. Saw the da Vinci, many Van Gogh's, and lots of other paintings before being drawn to Monet. And as we looked around at several Monets, it became obvious that he had a special artistic style that would lend itself well to replication by our robot and Deep Learning algorithms. In the end Hunter and I decided on the painting  the artwork on the left below - Monet's "Woman with a Parasol."

Now if we could somehow program our robots to capture and paint the wind across her face like Monet did. Wow, that would be amazing.