Context Paper

For my thesis I designed and implemented a new approach for applying style transfers to 360 pictures and video. Using contemporary and historical paintings and images from Google Street View, I created art that encourages viewers to appreciate what is beautiful or poignant about the modern world, right in front of us, and to see it with new eyes.

The spark that initiated my interest in this project is Ruder, Dosovitskiy, and Brox’s paper “Artistic style transfer for videos and spherical images.” I was intrigued by the idea of applying a style transfer to a spherical image because I had recently implemented an efficient 360 rendering algorithm to my open-source Processing library Camera3D. However, I was disappointed when I noticed some flaws in the 360 video provided by the paper’s authors demonstrating the technique. Based on my experience with the math behind 360 images, my curiosity led me to imagine other approaches that would work better.

Artistically I am inspired by the work of video artist Nam June Paik (1932-2006). Although his artistic medium is different from mine, his willingness to explore and modify the underlying electronics of television and video equipment motivates me and my work with machine learning and computational tools. I am also inspired by artist and neuroscientist Santiago Ramón y Cajal (1852-1934). His detailed drawings of neurons pioneered the field of neuroscience and earned him the Nobel Prize in 1906. It is because of his skills as an artist that he distinguished himself as a scientist and it is because of his work in neuroscience that he made a unique contribution to the art world. My ambition is that my background in technology and quantitative research and my new knowledge of the art world will empower me to have a similar impact.

There are some artists working with machine learning tools that I have been following and have learned from. The first is Mario Klingemann, a German artist and Google Arts and Culture resident known for his work with GANs. One project of his that I particularly like is Neural Glitch (2018). What I like about it is he “manipulate[s] fully trained GANs by randomly altering, deleting or exchanging their trained weights,” or in other words, interfering with the neural networks by changing connections and numbers. Instead of treating neural networks as sacrosanct he is tinkering with them just like one would with any other tool. I see a relationship between this and the work of Nam June Paik.

I am also influenced by the work of Anna Ridler. Her work Mosaic Virus (2018) generates images of tulips using a GAN. I’ve always been attracted to flowers, and I appreciate the conceptual meaning behind her work that links the displayed tulip to a financial instrument.

Next is Gene Kogan, who I think does a great job of teaching complicated machine learning subjects and making them accessible to beginners. I would like my project to be accessible to people less familiar with machine learning and programming and I will discuss with him ways of making that happen.

Julius Horsthuis is an artist who doesn’t typically work with machine learning tools but does work with math and fractals. He uses the rendering program Mandelbulb3D to create beautiful fractal animations. His work Recurrence (2017) combines fractals and style transfers and helped inspire my project in last year’s Project Development Studio class. His 360 videos are also inspiring.

Ben Fry and Casey Raes are the founders of Processing and are committed to making technology accessible to artists and creators. Processing was my gateway into deeper involvement with the creative technology world and it will always have a special place in my heart.

Much of my artistic research consisted of reading about artistic movements such as Impressionism and Fauvism. I wanted to learn more about these artists so I could make thoughtful choices about which of their works to use in a style transfer. I took several trips to museums to view work and took an online Art History class through Khan Academy. I also took watercolor and acrylic painting classes to get the experience of picking up a brush and representing an image on paper.