One of my main goals is to get machines to look at the world like a human. I make inferences about how we think about things from our perspective, and compare that to how machines are trained to look at data. My theory is that our thoughts, ideas are very low dimensional. They can be described by very few variables, in another sense they could be viewed as latent. Our observations of the world and actions in the world are not low dimensional. Our actions are based off of a latent representation of any given input data, this including observations and past observations and feelings. I think that a lot of our brain power goes into converting observations in the world, sounds, images, into a lower dimensional representation of what it means to observe these sounds, and images.
One paradigm that really contributes to the difference between how we generalize about the world and how computers do, is where exactly we observe how wrong we are, our loss function. In deep learning, let’s take a classifier for instance, we tell the classifier how wrong its predictions were on the last layer of its multi-layer perceptron. The only feed back the classifier gets is how wrong it was in its prediction, then we back propagate through the network and change all of the weights in a way that most quickly decreases this loss function. This is back propagation by gradient descent, and when training a large network the gradient sort of vanishes as we approach the beginning layers. This is known as the vanishing gradient problem, it can also be seen in things like recurrent neural networks, where you have to back-propagate through time. Essentially what I am trying to get at, is that the only feedback we give our networks is the feedback at the layer of classification, where a human decides what’s right or wrong. Our brain is of divine design, something more intelligent than a human created us and the way we think and process information. This is why advancements in science, particularly in deep learning are almost beautiful. Advancing in deep learning means that we learn better and better ways to generate images, to classify data, in ways that a human would.
Thinking about this problem led me to my idea of a Dream-GAN. We know about the deep dream deep learning project by open ai, that fascinated me more than anything else at this moment. How we could visualize learning, how I could see how the network is generalizing a specific phenomenon. I thought this was particularly cool with a classifier trained on recognizing animals. Cats, dogs, rabbits, foxes, and many more animals were the subjects of classification for this convolutional neural network. What’s interesting, and where DeepDream came in, is when we give it any old image, let's say just a landscape, then we ask the network to change the image in order to make specific neurons light up, fire. A highly abstracted neuron, or layer, is one close to the classification layer, which recognized higher order patterns in the data. When asked to maximize the probability of these neurons firing given an image iteratively, these higher order patterns actually start emerging from the image! Given that this network classifies images of animals, you would expect these higher order patters to be the network’s generalization of what a dog is, or what a rabbit is. You can either choose to maximize an entire layer or you can choose to maximize a single neuron, who’s task might be to recognize a single feature. Now you can do this with the highly abstracted neurons and layers, or you can do this with lowly abstracted layers of neurons. These lower abstractions are representative of simple features in data, in this convolutional net, it might be lines and edges. When asked to maximize these layers you get a change to the image that almost looks like a painting from the Impressionism era of art.
What was interesting to me, was that now we don’t only have the last layer to judge. In fact with this tool, now we can see what exactly every step along the abstraction does. Still as humans, I believe that we are best suited to not meddle in how the network decides to interpret and find patterns in data. This is why I have not acted on my ideas for Dream-GAN yet, because I believe that I have no business telling the computer how to abstract data to come to a conclusion, because I am ignorant as to how I do the same process myself. But let’s say for a thought experiment that somewhere out there, there is a secret as to the perfect way to abstract things, would it be possible to meddle in the classifier’s ‘thought process’? I believe this solution, this higher order truth, to be out in the universe somewhere, but inaccessible to us. Let’s take a classifier C, a convolutional neural network who’s task is to map input images to their correct label. C has 10 layers in total, 8 of which are hidden, containing the abstractions of the input data. Flipping C upside down, and changing the input to increase the probability of neurons on a given layer firing changes C from a classifier network to a generative network. Let’s say that there was some observable thing, phenomenon, idea in the universe from which we can physically observe and be sure that it is indicative of a given level of abstraction. A discriminator, D, will have the task of discriminating between the output from the generative classifier and between this perfect unknown abstraction. The generator will then learn to tune its weights in a way that approaches this unobservable perfection. Here’s the trick though, we don’t want the classifier to forget how to recognize objects, so the classifier will interchangeably learn how to classify phenomenon and have the weights approach the objectively correct abstract representation, in its attempt to fool the discriminator.
Why can we not be this gauge is a question you might ask. It’s because we do not know, we cannot know how to compute a loss function between what the layer of abstraction should be compared to how it is. We don’t know what the layer of abstraction should look like, there is no optimization formula. Now you might say, “Jack, why? It sounds so complicated and unnecessary, and it has too much computational complexity?” Why exactly I had this thought was because I understood that we don’t form our loss functions in our brains the same way that we do with deep learning tasks. It really DOES matter how thoughts are formed inside of our brains, and it's not a completely random process as back propagation might suggest. I think that there really is some higher order truth, idea, force that is beyond our understanding that guides our thought process in the right direction. What I have thought up is not perfect, probably very messy and flawed, but it is a mechanism. A mechanism for judgement where humans cannot, we can only judge if the classifier was wrong or right, but this mechanism might be able to judge if the thought process, the abstraction process is wrong or right.