Artificial Intelligence at the Top of a Professional Sport

The lights dim and the music swells as an elite competitor in a silk robe passes through a cheering crowd to take the ring. It’s a blueprint familiar to boxing, only this pugilist won’t be throwing punches.

OpenAI created an AI bot that has beaten the best players in the world at this year’s International championship. The International is an esports competition held annually for Dota 2, one of the most competitive multiplayer online battle arena (MOBA) games.

Each match of the International consists of two 5-player teams competing against each other for 35-45 minutes. In layman’s terms, it is an online version of capture the flag. While the premise may sound simple, it is actually one of the most complicated and detailed competitive games out there. The top teams are required to practice together daily, but this level of play is nothing new to them. To reach a professional level, individual players would practice obscenely late, go to sleep, and then repeat the process. For years. So how long did the AI bot have to prepare for this competition compared to these seasoned pros? A couple of months.

So, What Were the Results?

Normally, a professional Dota 2 game is played on a stage with 5v5 teams. This was the bot’s first competition, and the AI only had a couple of months to learn how to play Dota 2 completely from the ground up. It seemed more fair to start off simple with 1v1 matches. Those first matches were against [Dendi], one of the top players in the world, who lost to the bot shown below in the first match within about ten minutes, resigned in the second match, and then declined to play the third.

The OpenAI team didn’t use imitation learning to train the bot. Instead, it was put up against an exact copy of itself starting with the very first match it played. This continued, nonstop, for months. The bot was constantly improving against itself, and in turn it would have to try that much harder to win. This vigorous training clearly paid off.

While the 1v1 results are stellar, the bot has not had enough time to learn how to work in a cohesive manner with 4 other copies of itself to make a true Dota 2 team. After the roaring success of the International, the next step for OpenAI is to form an ultimate 5 bot team. We think it will be possible to beat the top players next year and we’re eager to see how long that takes.

What Does OpenAI Like to do When it is Not Busy Crushing Video Game Competition?

OpenAI has worked on a number of projects before the Dota 2 effort. They explored the effect of parameter noise to learning algorithms which has proven to be advantageous across the board. During exploratory behavior used in reinforcement learning, parameter noise is used to increase the efficiency of the rate at which agents learn.

The left diagram represents action space noise, traditionally used to change the likelihood of each action step by step. The right diagram represents the newly implemented parameter space noise:

“Parameter space noise injects randomness directly into the parameters of the agent, altering the types of decisions it makes such that they always fully depend on what the agent currently senses.”

By adding this noise right into the parameters, it has shown that it teaches agents tasks far faster than before. It’s part of a wider effort focusing on new ways to optimize learning algorithms to make the training process not only faster, but also more effective.

They are not done with Dota 2 either. When they come back with their five bot team next year, it will undoubtedly require a level of teamwork never before seen in artificial intelligence. Think of the possibilities. Will this take the shape of a collective hive mind? Will team dynamics among AI look anything like those of their human counterparts? This really is the stuff of science fiction being developed and tested right before our eyes.

Now, Why Might a Billion Dollar AI Startup Be Meddling in a Video Game Competition?

OpenAI is an open source company dedicated to creating safe artificial intelligence and working from a $1 Billion endowment established in 2015. On their website, they state that the Dota 2 experiment was, “a step towards building AI systems which accomplish well-defined goals in messy, complicated situations involving real humans.” The International was proof of concept that they could in fact implement AI that handled random situations successfully — even better than humans. It leaves us wondering if the next field AI dominates in won’t be something quite as trivial as a video game competition. It is notable that OpenAI’s chairman, Elon Musk, gave a warning statement directly after the victory:

“If you’re not concerned about AI safety, you should be. Vastly more risk than North Korea.”

This is not the first time that Musk has conveyed hesitations towards the upcoming dangers of our superior Dota 2 players. In fact, he has a history as a leading doomsayer:

“With artificial intelligence, we are summoning the demon. You know all those stories where there’s the guy with the pentagram and the holy water and he’s like, yeah, he’s sure he can control the demon? Doesn’t work out.”

It is clear why someone so worried about the future of AI has devoted his time and resources to a company dedicated to ensuring its safety for humanity with this advancing technology. But it almost seems paradoxical. Teaching an AI to compete better than humans appears to be marching that dreaded outcome one step closer. But at the same time, you can’t temper the advancement of technology by refusing to take part in it. The company’s approach is to make sure everyone can study and use the advancements they are making (the “Open” in OpenAI) and thereby prevent an imbalance of power presented if the best AIs of the future were to be privately controlled by a small number of companies, individuals, and state actors.

Earlier this year, Hackaday’s own Cameron Coward wrote up an in-depth article about the potential future of artificial intelligence. He delves into one of the hotly debated topics with this subject: the ethics of strong AI. Will they be malevolent? What rights should they have? These questions will be answered in the upcoming years — whether we want them to be or not. It is our job to make sure that the answers to these questions in the near future are not answered for us. OpenAI is debugging AI before it debugs us.