In transportation engineering, there is a phenomenon called user equilibrium. As engineers, we will change a transportation structure in a way that is optimal for us. The motivation, the objective, of the human driver will not change. Adding more lanes to a high-way may not necessarily decrease congestion on that high-way, it may just attract more drivers to the road and clogging it up again. The transportation network is changed to decrease traffic; but humans will reroute in ways the engineers might not have expected. William Alexander, in an email exchange, explained to me that there are two different equilibria in these systems - the one in which the network is satisfying its objective (maybe in a MARL environment), the other in which the users are all taking actions to maximize their respective objectives.
It is a bit psychological, our analyzation of people’s behavior in order to predict their actions, and it’s a bit out of the domains of computer science or civil engineering. I believe that if the network’s objective aligned with the objective of the individual cars, the network will operate with a higher efficiency. The problem is that people are greedy, and take actions that are selfish - not what is best for other people. When we are designing a system, we don’t want to make it only work for just one person, we would like to make it best for all people in it - we cannot change human behavior.The human behavior in the longer term will tend to mitigate winners and losers of games between intersections, driving the system towards user equilibrium. I believe that the correct way to mitigate this problem would be to ensure that there are no winners and losers, that all agents (intersections) in a MARL environment are of equal status and equal development throughout the process of training. My solution for this is self competitive policy optimization.
When there is a multi-agent system, each agent at every physical level has some ‘objective’ - some purpose to fulfill. When we look at a colony of ants, we see how perfectly they behave as a group, protecting the queen and finding optimal roots to food. The individual ant’s objective is not the same as the collective’s objective, but it is designed in such a way so it is contributive to the collective. What is essential to understand here is the ‘communication' between the collective and the individual - the checks and balances they place on each other. I think it is apt to describe the collective - the ant colony - as an organism just as we describe the ant itself as an organism. The colony, just as the ant, can adapt to changes and has an objective. The collective organism gives the individual ants structure, order, goals. The individual organisms - ants - operate with an appropriate, but limited level of autonomy.
Every organism has an inherent selfish need to survive, the ant has a need to survive, and the collective organism - the ant colony - has the need to survive. A complex relationship emerges when these objectives conflict, if the ant is being too selfish then the colony will not be operating at maximum efficiency. There should exist some reward structure acting as communication between the collective and the individual, such that if the individual is not contributive enough to the collective it should be punished. But on that same token, the ants need a certain level of autonomy in order to ensure collective efficiency. Why is this? The collective does not have infinite intelligence as to how the individuals should behave, and it is not optimal to model it as such. The optimal efficiency of the collective organism relies on an appropriate level of autonomy of the individual.
Maybe it is more relatable to describe this relationship between the individual and the collective with societies. Every human shares similar wants and needs; if you believe in Darwin’s theory of natural selection, then one of the main needs is to pass on genetics, everything else depends on that. The objective of humans is complex, we have the drive to reproduce, the drive to be successful, the sex drive, a drive to bond, a drive to learn and much more. These are all objectives of the individual humans - you and I. I would argue that a reward structure is very much present between the individual and the collective in societies. If an individual over indulges in a personal objective that is not contributive to the collective, they will feel a societal pressure to conform. The collective’s vitality is treated by the individual’s overindulgence in personal objective, consequently the collective will punish the individual. Sorry for this rather graphic description, it is the most clear way to describe this relationship. Human individuals have a sex drive, overindulgence of that sex drive can lead to rape, which exists across almost every society in the world. Obviously, sexual violence is not contributive to the prosperity of the collective organism. This is why sexual violence is not accepted as an okay thing to do to another human. The society can be looked at as an organism in this sense, it has its own will to survive. This particular example of a human overindulging in a particular objective means that the vitality of the collective organism is threatened. The individual who overindulges will be shunned, maybe banished from society.
It is weird for us as humans to think that something bigger than ourselves has a will. That a tribe of a thousand people has its own collective will to survive and thrive separate from our own. The pretense to all of this is that everything has a will to continue continuing. The other pretense is that the relationships between all things is complex and interdependent. These wills may cooperate or they may conflict, more often than not they will conflict. The third pretense is that everything has autonomy. Things, life, relationships can be modeled as hierarchical. Every living thing both depends on things and have things that are dependent on them. In other words, every living thing controls things, and is controlled by things.