The fundamental Problem with Supervised Learning

The fundamental problem with supervised learning is that the learning of the optimal latent representation of the input data and the segmentation of that space for the purpose of classifying is the same. The abstraction of the input data is an abstraction to the ends of the output, it twists and molds the input data to form the shape of the output. Just compare for a second how unlike that is to real learning, or at least how we perceive it to happen within our minds. I would make the claim that we do not learn by abstracting input data, but learn by making sense of abstract data referred to by an observation. What I mean by this is that how we perceive the world and how we think about the world - the input - is already sufficiently abstract before we go around hypothesizing and making sense of it. That is what I believe is limiting us on this quest to understand intelligence - the separation of abstraction and classification.

The difficult situation we are in now is that abstraction of data is related to the usefulness of that data - but useful to what ends is the appropriate question. Our goal in AI is to eventually produce an AGI capable of doing the amount of tasks a human can, better than a human can. So taking into account the goals and tasks of a human may be an appropriate thought process. Another helpful way to think about this problem is through language. We tend to think of language as an input, which we process and make sense of. But language is an abstraction of reality in and of itself, and a quite intelligent one at that. The purpose of language is the communication of ideas. Language is like a tool box, its contents can be used to solve many problems humans encounter - and it is important to understand that it was developed for this purpose. This leads me back again to the fundamental problem with supervised learning. I believe that most supervised learning architectures are attempting to learn both a language to describe a specific phenomenon, and problem solving techniques. I have no real evidence to back this previous claim up with, but it is true that the abstraction of the input data and the classification function are identical. I think this is what makes scaling inefficient. When the language for describing a phenomenon is inseparable from given instances of a problem, it is impossible to predict new information too far away from training data. It is inefficient to learn a new language every time we see new input data, a sufficient language should be generalizable to new information.

Language has the property of being both generalizable and abstract. It is a framework that is flexible enough to describe unknown phenomenon while still being simple enough to be understood by a group of individuals. Abstract and observable can be thought of as opposites for now. An abstract representation of an observable phenomenon is a way to reference or describe the observable phenomenon (the abstraction often being of a lower dimensionality). In programming, an abstract language like python is one that is sufficiently separated from the hardware executing the program. Abstraction comes with ease to manipulate and process information, but a proper abstraction is difficult to achieve - one that is relatively simple but has a lot of capability. If python were built for only solving one type of problem, it would not be a good language, regardless of how easy it would be to code. In a way, language is the meta-heuristic to a set of problems. Language is a sufficient abstraction of the world not for any particular problem, but to solve particular problems. Language is a formalism for connecting independent problems using the same framework. Without developing a language independent of the problem at hand, the development of language becomes the problem, and is in-expandable.