Auto-encoders use two features. The first is the reconstruction element, which ensures that the data in an encoded state is still representative of the original state. Auto-encoders have been one of the main focuses of unsupervised deep learning in the past decade. They are important, useful, and interesting because of their ability to compress data. Data compression sounds technical, all it means to me is being able to express high dimensional data in a way that is lower dimensional.
To do this, to represent high dimensional, complex data, in a lower dimensional or simpler way means that there must be some patterns in the high dimensional data. If there aren’t any observable patterns in the high dimensional data, there is no hope of being able to have a lower dimensional representation of that same data. Another term for the lower dimensional data are the latent variables, which are decoded to reconstruct the original data. The structure of an Autoencoder looks like an hour glass, information is passed from the top to the bottom of the hour glass. The two wide parts at the top of the hour glass would be some input data, high dimensional, as it passes down, it is forced into a lower dimensional form in the bottleneck of the glass. After this, the information is reconstructed back into its original form.
So why would something like this be useful? Obviously we have the idea of data compression, and that would improve the transmission of information, given one would have to send much less of it. Another cool innovation, fairly useful, is the denoising Autoencoder. Instead of reconstructing the input data to a tee, the network is encouraged to produce an input that is a ‘cleaned’ version of the input. The case is that there are pictures, but they are blurred and you might want to clean them up a bit. With your computer you’re reading this on, you could go into photoshop or whatever and increase the blurriness on a picture. Do this with a ton of pictures of numbers, literally almost a ton, about 1200. Keep the original copy too! Have the input of your autoencoder be this blurry image. Calculate the reconstruction loss between the output of the decoder and the unblurred original image. Now, given a new blurry image, the autoencoder can output a cleaned up version.
Looking at it this way, one might say that this is a supervised learning problem. I would pretty much agree with that claim given just this information. After all, there is some function which transforms blurry images into non-blurry images, and the autoencoder approximates it. That was the whole point of supervised learning… right? There is some function and we use a neural network to approximate it. My belief is that this de-noising autoencoder is somewhere between being unsupervised and supervised. Unsupervised has a loose definition, one that really does not make much sense when you think about it. In a normal or denoising autoencoder, you would say that it is unsupervised because how it compressed information and what each of the latent variables relate to in the context of the input data is not configured by our intelligences. I think a more precise way to define unsupervised learning, is that the most compressed representation of the input data is NOT the level of judgement.
It’s really more of a scale, a spectrum between supervised learning and unsupervised learning. A classifier algorithm is an obvious example of a supervised learning, let's say this algorithm decided whether someone has a malignant tumor. Now defined as a function, the neural network learns the mapping from images of tumors to a bit sized output of malignant or benign. Normally, deep neural nets are depicted on paper as the information flowing through the network from left to right. Imagine a triangle, with a huge left side and a small right side. ‘Learning’ means abstracting the data until it reaches its lowest dimensionally, in this case, being malignant or benign. In an autoencoder, the information from the input gets progressively more abstract until it reaches the bottleneck of the architecture, after this, the information becomes less abstract and higher dimensional. In a more supervised learning setting, the purpose of the abstraction is to abstract the input data into a given definition of that input. Again with the classifier, the input data is manipulated, abstracted into a definition for that data. One important note is that abstraction and dimensional reduction go hand in hand, to be more abstract is to define a complex system in a simpler way. To reiterate, supervised learning occurs when the input information is abstracted to the definition of that input.
In the denoising Autoencoder, the function mapping blurry images to normal images is still being fit to a neural network. The difference is that data from the blurry images is not being continuously abstracted to the point of the output. The data from the blurry images continue to be abstracted until you hit the bottle neck, the encoded state of the input data. After this, the data becomes less abstract until it reaches the state of evaluation. Although there is this function being fit, I would not call the learning that is occurring supervised learning, because function isn’t fitting the abstraction of the data.
For a standard autoencoder, the reconstruction error ensures the preservation of the input data in the encoded state. Moving onto more complex versions of the autoencoder, this feature is still essential and will always be a part of the objective function.