OpenAI, a research lab co-founded by Elon Musk, has developed a new breed of AI agents that are capable of playing Dota 2 in 5-on-5 matches. While OpenAI’s new bots have only been able to beat amateur and semi-professional teams so far, the lab is now looking to challenge a team of pros at The International 2018 (TI8) this coming August.
The team of bots go by the name of OpenAI Five, referencing the number of neural networks working together in the team, and has been playing roughly 180 years worth of gameplay against it self every day using self and reinforcement learning.
The OpenAI Five represent some of the latest advancements in AI technology, which impressed even Microsoft Corporation founder Bill Gates, who lauded the lab’s work in a tweet:
#AI bots just beat humans at the video game Dota 2. That’s a big deal, because their victory required teamwork and collaboration – a huge milestone in advancing artificial intelligence. https://t.co/UqIUhh9xFc
— Bill Gates (@BillGates) June 26, 2018
The capabilities of AI systems and computer programs have long been tested against the human mind in games like Chess and Go, where the likes of Deep Blue and AlphaGo defeated world-class players and champions in those games. However, a video game like Dota 2 is a different beast altogether.
Widely agreed as the most macro-intensive of the big three Multiplayer-Online Battle Arena (MOBA) genre, as playing Dota 2 at a high-level play requires not only requires fast hands or twitchy reflexes. Frankly speaking, mechanical skill can only take a player so far. To become a successful Dota 2 pro player, a deep understanding of the game’s strategies and gameplay nuances are needed.
“It took me like 8 years to learn how to play at this level, to learn about all these [pro] strategies,” said William “Blitz” Lee, a former pro player turned coach and broadcaster, who helped the lab playtest its bots.
While the AI may have the advantage when it comes to mechanics (they average around 150-170 actions per minute and have a reaction time of 80ms), they cannot trump humans when it comes to lateral or creative ways of dealing with problems in-game. For example, Dota 2 has gameplay mechanics such as the the ‘fog of war’ which hides information from players and prevents them from observing the entire playing field and making decisions based on that like in Chess.
If it takes human players like Blitz more or less 8 years to learn about not only the basic mechanics, but also more advanced gameplay tactics, then it would require a herculean effort by the AI to learn Dota 2’s myriad intricacies.
OpenAI employs some very impressive rigs to allow its bots to do just that. The five neural networks train through a scaled-up version of Proximal Policy Optimization, which are all running on 256 GPUs and 128,000 CPU cores. The setup allows the OpenAI Five to accrue what the lab estimates to be around 900 years (180 years for each member of a team of five) years of experience every day.
“The bot learned the game from scratch by self-play, and does not use imitation learning or tree search. This is a step towards building AI systems which accomplish well-defined goals in messy, complicated situations involving real humans,” OpenAI said in a statement.
The lab used the same setup, albeit in a much smaller scale, when it developed a 1-on-1 bot that defeated world-class Dota 2 pros like Artour “Arteezy” Babaev, Sumail “SumaiL” Hassan, and Danil “Dendi” Ishutin back in TI7.
While the 1-on-1 bot took pros by surprise during its debut, players quickly found ways to defeat it when OpenAI made it available for challenges following TI7. Even if the lab’s run with its 1-on-1 bot was successful, extrapolating that to the full scope of Dota 2 games, with 5-on-5 matches featuring a virtually limitless number of hero combinations, are a much more complicated matter.
To start, OpenAI instituted a reward system for its bots on an individual level, which consist mostly of metrics humans players use to decide how they’re doing in the game, like kills, deaths, assists, last hits, net worth, so on. While the bots started out with playing to get the best kill-death-assist score, much like any human player, they eventually learned how to balance these out with the objectives needed to win the game.
While the OpenAI Five do not explicitly communicate among themselves, teamwork is controlled by a hyperparameter the lab dubbed as “team spirit”. Team spirit ranges from 0 to 1, putting a weight on how much each of the bots’ heroes should care about its individual reward function in contrast to the average of that of the team.
With that in mind, OpenAI also placed certain restrictions on its bots under which they, and their human opponents, would play, which include:
- Mirror match of Necrophos, Sniper, Viper, Crystal Maiden, and Lich
- No warding
- No Roshan
- No invisibility (consumables and relevant items)
- No summons/illusions
- No Divine Rapier, Bottle, Quelling Blade, Boots of Travel, Tome of Knowledge, or Infused Raindrop
- 5 invulnerable couriers, no exploiting them by scouting or tanking
- No Scan
The set hero restrictions may make the game very different from how Dota is played at a professional level (i.e. Captains Mode drafting from all 100+ heroes), but the difference from regular “pub” games (All Pick / Random Draft) is smaller.
The research lab let the OpenAI Five play against an array of opponents, with skill levels spanning from amateur to semi-pro.
The results proved to be staggering, as the different versions of the OpenAI Five adapted and learned from its human competition.
“The May 15th version of OpenAI Five was evenly matched versus team 1, winning one game and losing another. The June 6th version of OpenAI Five decisively won all its games versus teams 1-3. We set up informal scrims with teams 4 & 5, expecting to lose soundly, but OpenAI Five won two of its first three games versus both,” said the research lab.
OpenAI also noted that their bots used strategies not unlike those commonly employed by professional teams. These include sacrificing their own safe lane (a team’s easily defensible side of the map) in exchange for controlling that of their opponents, pushing the transitions from early- to mid-game faster than its opponents through ganks and tower pushes, and deviating from current playstyles, such as giving support heroes (which usually do not take priority for resources) lots of early experience and gold.
However, the bots still exhibited some weaknesses. The lab noted that the bots were not particularly good at last-hitting, which Blitz estimated to be around the average for Dota players. Something as simple as failing to get the killing blow on a creep can have big repercussions in a game. Dota pros are well-versed in skillfully last-hitting to not only gain an advantage in terms of gold and experience, but also to control the state of the battlefield to get superior positioning in the lane or make it easier to set up a gank or tower push.
However, Blitz also noted that the bots’ objective prioritization are akin to common professional strategies. Gaining long-term rewards such as strategic map control in Dota often requires sacrificing short-term rewards such as gold gained from farming, since grouping up to attack towers takes time.
“The teamwork aspect of the bot was just overwhelming. It feels like five selfless players that know a good general strategy,” said Blitz.
But will that be enough to defeat a team of world-class professionals? Most probably not. Especially if the matches are played in multiple games, as pros always adapt and innovate over the course of multiple-game series in tournament play.
If anything, OpenAI’s challenge against a pro team would most likely end up like its 1-on-1 bot. While players were surprised and defeated in the first few encounters, they eventually adapted over time to soundly defeat the bot.
Even if the bots do end up losing, what OpenAI and its bots have already done is a significant feat by itself, not only in the field of AI technology but in gaming as well.
“Games have really been the benchmark [in AI research], these complex strategy games are the milestone that we have all been working towards because they start to capture aspects of the real world,” said OpenAI cofounder and CTO Greg Brockman in a statement.
But what would happen if the bots win? Would their victory in Dota 2 push them to take their advancements further, using the skills they acquired by playing Dota to go all Skynet on us and enslave humanity? Will we have to battle the AI in a Dota 2 tournament to save civilization as we know it?
Kidding aside, the lab seem are confident and proud of its AI team, whatever the result may be. Before facing pros at TI8, OpenAI will benchmark the bots’ progress with a showmatch versus top players on July 28th on its Twitch channel. For a deep dive into the research lab’s process, check out their blog.
The International 2018 will be held in the Rogers Arena, in Vancouver, Canada from August 20 to 25. Not only will you witness the best Dota 2 teams in the world duke it out, but you can also see if AI can beat humans in video games or not. Now, ain’t that something.