Digits that are not: Generating new types through deep neural nets
Authors’ comments (Akin Kazakçi, Mehdi Cherti, Balázs Kégl)
Computational creativity has been gaining increased attention in the machine learning community and in the larger public. The wave started by the freaky psychedelic images of deep dream, generated by saturating neurons starting from a natural image (an artificial “migraine”?). Style transfer is another interesting example of a neural net exhibiting artistic traits, although it arguably falls short of what we normally call creative. Closer to the concept, Google’s new Magenta project released their first single, a tune generated by a neural net. Despite the obvious interest (indeed, it is hard to imagine how one can think about creating AI without the capacity of designing novelty, solely on the basis of prediction), these activities remain at the fringes of machine learning research. There is also little communication between ML and the computational creativity community which have been working on both the conceptual underpinnings of creativity and methods to generate creative objects.
In this paper we introduce a definition that goes beyond the traditional likelihood-based approaches by requiring the net to use encoded knowledge to generate things it has never seen. More precisely, we use a deep generative autoencoder, trained on an image data set to generate new types of hand-written symbols. The paper is published at ICCC’16, but it is written both to the (deep) machine learning and to the computational creativity communities. Both domains have their notions of what a generative model is, and while these definitions can be reconciled, examining the two different angles opened up a direction which may be interesting to both communities. Our main point is that the true generative power of deep nets can be revealed by exploring unknown object types rather than by recognizing or generating known objects that are similar to the training data.
This is a short commentary to underline our messages addressed to the machine learning community.
Stop fixating on likelihood
The main bottleneck for a scientific study of creativity is that we have no preconception about the value of the objects we wish to generate. Artists play with knowledge, ideas, concepts, but most importantly, they invent their own value function. This fundamental issue of problem definition is very different of what we used to deal with in ML, where we are usually looking for unknown solutions for well-defined problems.
The goal of generative modeling in ML is to generate objects that are different but not very different from known (training) objects. The problem is commonly framed as probabilistic sampling from a fixed but unknown distribution. Objects are then evaluated by computing their likelihood under this distribution. The trouble is that the likelihood is unknown, and the development of surrogates is guided by heuristics about what it means to be “different but not very different”. Because of this, the concept of likelihood, in the case when all we have is a sample, is more of a self-imposed straitjacket than a constructive idea. Once we get rid of our narrow notion of likelihood, it turns out that our existing algorithms can also generate objects that are “quite different yet plausible”, exhibiting a dim sign of creativity.
The 2D projection on the left shows that an autoencoder can generate new digits, but also other objects which look like digits in another world but whose likelihood, under the narrow ML goal of generative models, is presumably low. In the paper, we deliberately tuned the autoencoder to become more creative-explorative and to come up with new types of objects. At this point we used a human in the loop to validate what a new type is. Finding a way to automatically discover the value of these objects is the crux of computational creativity. At this point we have no solution, only a playground in which this question, novel to the ML community, can be studied.
Focus on the “what”, not the “how”
In a more general sense we would like to see more papers on the question of what we should do with deep nets instead of how to do it. Our culture problem solving is inherently biased towards incremental improvements on techniques that solve known problems. While pushing the state of the art is of course important, major breakthroughs usually come from inventing new uses. The obvious example is AlphaGo, usually seen as a great achievement in solving a hard problem whereas one could argue that their main contribution is to invent the problem at the first place: noticing that games, with infinite data and non-trivial tasks are a sweet spot for making deep learning shine. Creativity itself is as much about inventing our goals as solving them. Our argument has two sides: we push for more creativity in ML in inventing new problems while also going for algorithms that invent their own goals.
