Google engineers have taught artificial intelligence to play Starcraft II. Now there are only a few people in the world who can compete with him

The engineers of DeepMind (since 2014 - the Google unit) presented the final version of their AlphaStar program - artificial intelligence that can play the classic Starcraft II strategy. Back in January 2019, AlphaStar played at the level of advanced amateurs, and today the improved AI is included in the highest 0.2% of players in Battle.net and on equal terms competes with the best of the best. Medusa is dealing with how long the human Starcraft II remains and what will happen when the classical strategy will finally fall under the onslaught of AI.

What is this program and why is it being developed?

AlphaStar is the heiress of Alphago, a program that defeated humanity in the game GO. Like Alphago, this is the development of the DeepMind team, headed by Demis Hassabis. It is one of the authors of the new article in Nature, which describes the AlphaStar device, however, the leader of the new project was another artificial intelligence specialist - who came from Google Brain Oriol Vinyals, who participated in the development of machine translation and speech recognition services for the American company.

Both Google Brain and DeepMind are primarily engaged in experimental projects without any direct practical significance. In addition to game AI systems, among the well -known DeepMind projects, one can note the Wavenet neural network synthesis and undertaking in the field of medicine - for example, an experimental system for diagnosing images in ophthalmology. Google Brain engineers are often engaged in projects that are more close to art than research experiments.

As Vignals and co -authors indicate in their work, the developments that allowed us to make a successful bot in Starcraft can be useful for managing robots, personal assistants, autonomous transport and generally influence the future of AI.

How is Starcraft different from chess and go?

Despite the general ideological kinship of projects and almost one team, AlphaStar is very different from Alphago - just as these games differ from each other.

Firstly, Starcraft refers to a completely different class of games-games with closed information. The difference is that in games with open information - chess, checkers and backgammon (long -defeated AI), th (defeated by him recently) and many others - each of the participants knows everything about what is happening on the game field. Chess players always see the board and know who has many figures, players in the GO completely see Goban (playing field), and so on - only in such games only how the enemy behaves.

But in most card games and in Starcraft, players do not have all the information about enemy information. In such games - with closed information - modeling the future development of the party “on the forehead”, that is, overwhelming, is usually much more difficult than in games with open information - the share of uncertainty in them is much higher.

Of course, not in any games, even with open information, you can calculate all the future moves - for example, in chess this is still possible, but in th - no, there is the number of permissible combinations of stones on the board exceeds the number of atoms in the universe. However, the openness of the information still gives the computer a certain advantage - even Alphago used to defeat the ability of neural networks to conduct a tactical analysis of the current situation “by eye”, but also the good old busting of future moves .

Starcraft is very different in this regard - it is almost useless to calculate the actions of the enemy there. Therefore, AlphaStar does not even try to do this - instead, the program learns to find a victorious strategy, focusing on the statistics of already played by strangers and his own experience.

Compared to go, Starcraft has several more important differences: this is a real time game in which all players act simultaneously, and not alternately. In it, the outcome becomes clear after many thousands of small decisions (and predicting who now has an advantage is very difficult - because it is not known how the enemy plays). Therefore, in Starcraft, to teach AI to distinguish the right moves from the wrong, focusing only on the outcome of the match, it would be almost impossible - for this it would be necessary to play for many years, if not a millennium. The training with the reinforcement that was used to create AlphaStar is a technique that has been known for several decades. Now she has high hopes. Google DeepMind, for example, almost completely relied on training with reinforcements in combination with deep training on neural networks. Generally speaking, these two techniques - Deep Learning and ReinForcement Learning - are quite different, they appeared and developed independently of each other. But now they are often trying to unite them in the same system and there are great hopes that they should work well together.

However, in the last one and a half to two years there is some disappointment in this area: training with reinforcements turned out to be a very complex methodology, attempts to apply which are not always successful to real tasks. This does not mean that it is time to be disappointed in this method. Just the opposite - you have to try again. It seems to me that this is the right direction, but it may be a task that we solve is actually more complicated than it seemed at the beginning.

As for the role of computer games in AI research (chess, GO, Dota-2, and now Starcraft), they are now used as a convenient training ground, where game convention and rules help to create something working. Real life and any business assaults, of course, are much more complicated. But if you try to imagine which of the real tasks has the greatest chances to benefit from research and application of training with reinforcement, I would say that this is robot training. Look, for example, at the latest developments of the famous Boston Dynamics - all these jumping, running, making somersaults would not be possible without the use of modern reinforcement teaching methods.

What success did artificial intelligence achieved earlier?

For the first time that Starcraft took up DeepMind, became officially known in 2017. Then Google, together with the developers of the game from Blizzard, created a special tool (API), with the help of which II could interact with the game, and through it with other players.

After that, a lull followed, and in January 2019, DeepMind introduced the first version of AlphaStar. This, the first version of AI, trained only on the basis of the analysis of the games of ordinary people, is called AlphaStar Supervized. She managed to win a total of eight matches from professional e -sportsmen, included in a hundred of the strongest players - German Dario Vunsh ( TLO ) and Poles of Grgezha Kominch ( Mana ).

What happened? What are the results?

Now the result was significantly improved. The AI ​​version, which was called AlphaStar Final, by the points that reflect skill in the game (MMR), grew one and a half times, as a result of which it managed to reach 0.2% of the best players in the Battle.net system.

It is important to note that it was possible to achieve this even after the authors eliminated the last advantage of the computer over people - back in January, the algorithm could simultaneously see everything that happens on the open part of the card, but the new version received information about the game field through the area (virtual) of the monitor - like all other people. At the same time, the computer was limited by the number of actions that it could perform per minute. The restrictions on this parameter, as well as on the delays between action and the result, were introduced by the authors of the algorithm in order to evaluate the capabilities of AI in strategy, and not simple computational power.

All these results were achieved with the help of a new approach to training, which can be used far beyond the boundaries of e -sports.

How did you manage to train a neural network that plays so well? Who taught her?

The training was divided into two stages, very different in mechanics of training. At the first (he is called “training with the teacher”) AI analyzed the games of ordinary people and studied on the basis of observing simple actions - the construction of units, their movements, an attack on enemies, and so on to predict those actions that are associated with victory. At the second stage, there was a turn of training with reinforcements - AlphaStar played with himself, while favorable strategies were encouraged, and those that lead to a loss were used less and less.

The main difficulty that the authors of AlphaStar faced at this stage lies in the ideal balance of forces that Starcraft is famous for. There is no universal strategy in the game that would lead to a win in all or in most cases - its AI could quickly learn and imitate it. The situation, as the authors note, rather resembles the game “Stone-Little Fighting Little Man”, when there are several strategies, each of which is strong against the other, but not against all at once.

When training with reinforcement, that is, when playing a computer with yourself, in the case of such games, a cycle usually occurs. Conditionally - if earlier the computer threw a stone, then then it will more often throw out paper, and then scissors and so on. As a result, instead of stable growth in skill, an endless cycle occurs, from which AI cannot get out.

To solve this problem, there is a technique in which the latest version of the AI ​​plays a series of games not against itself, but against all other previous versions - most of them are weaker, but if there are those that always win, then the new version learns to cope with them and getting hung up.

However, to enter the Higher, Grandmaster League, this was not enough. A large number of possible strategies excluded the possibility that AI will guess to them independently and check the possible response of the opponents. Then the authors of the new article resorted to two tricks.

Firstly, they added an “support on tradition” element to the AlphaStar architecture-the ability not only to study on their own, but also to focus on the statistics of how living players usually act in this situation.

Secondly, the authors decided to teach AI not individual, but in a collective way-different versions of AlphaStar formed a league in which different roles were assigned to different “players”. In addition to the “main composition” in the league were “helping players”, the task of which was to come up with the most creative and unusual attack strategy for the main players. Their average number of victories was small and even slightly reduced as they studied, but sometimes they invented strategies that were very successful against individual strategies of the main players. The presence of such assistants allowed to avoid too narrow specialization for the main players, which people against an electronic enemy could use.

Approximately this method, the authors trained two main versions of the system - weaker (27 days of studying on the Google development tensor processors) and stronger (44 days of training), which became the final version - AlphaStar Final. To understand how well they play, both versions (as well as several others, deprived of individual components) were released into the Battle.net arena in July 2019, where several dozens of games anonymously played anonymously. AlphaStar best succeeded in the game for “high -tech” protoses - on this race the system managed to reach a rating of 6275 MMR.

What do experts think about this?

Alexey "Alex007" Trushlyakov, commentator of e -sports competitions, video blogger, specialist in Starcraft

The elite (Grandmaster) League includes the 200 best players of the region. Accordingly, having entered the elite league on the European server, AlphaStar was among the 200 best players in Europe, which, from a formal point of view, is certainly very cool.

On the other hand, no hasty conclusions from only the fact of access to the elite league should not be made. Firstly, in Starcraft this status is submitted by many amateur players, and the gap between players from the second half of the elite league and the real pros is the same as between the grandmaster of the international caliber and the candidate for master of sports in chess. Secondly, in a situation where you play anonymously (as they say in Starcraft, under Barcod, that is, an account of the type of lllllllllll, about which the opponent knows absolutely nothing) and against random rivals, efficiency can demonstrate completely non-passing strategy at a professional level. Accordingly, the fact of access to the elite league speaks only about one thing: playing anonymously, AlphaStar knows how to beat most of the amateur players in Starcraft II in one way or another.

In order for Starcraft's game community to recognize its defeat in the battle with artificial intelligence, you need a full -fledged match of AI agents against the world team, which will include the best players of both Korea and the rest of the world. An excellent candidate for the role of the “last stronghold” of mankind is the current world champion Serral , who will try to protect his trophy in the playoffs of the new World Cup on November 1.

On the other hand, not a single e -sports discipline has yet suffered from success in the development of AI, so we will continue to follow any successes of DeepMind, Openai and others like them, sincerely wishing them success and interesting results.

For example, one of the main driving forces of interest in the development of AI on the part of the game community is an abundance of opportunities for training. The entire Starcraft's game community has a hood and cherishes the idea of ​​the emergence of new strategies and a new “meta” that could open artificial intelligence with its strategies and its approach to the game. Imagine a kind of smart game assistant who at each stage of the game tells you that you are doing wrong, where you have deviated from the optimal strategy and what is the weakness of your current position, and the commentator receives the probability of the victory of each player in real time confrontation. The development of game AI can very much change the appearance of gaming and e-sports, but so far we are still not so close to this reality.

Given that Starcraft at the highest level is a game of far from only the speed of the brain and speed of clicks, but also the maximum depth of strategies, tactical techniques and a kind of “mind games”, I believe that AlphaStar needs to take not just a step, but a huge jump forward. I have no idea how much DeepMind will be needed to take this step and whether they want this - it is quite possible that they are ready to stop right now, and then the Starcraft community will be able to proudly tell what we resisted under the onslaught of AI. At least when AI finally began to play honestly.

Konstantin Vortsov, professor at the Department of Intellectual Systems MIFT, teacher of HSE and Shad:

The reinforcement training that was used to create AlphaStar is a technique that has been known for several decades. Now she has high hopes. Google DeepMind, for example, almost completely relied on reinforcement training in combination with deep neural networks. Generally speaking, these two techniques - Deep Learning and ReinForcement Learning - are quite different, they appeared and developed independently of each other. But now they are increasingly trying to unite them.

However, in the last one and a half to two years there has been some disappointment in this area: training with reinforcements turned out to be a very complex methodology, attempts to apply which are not always successful to real tasks. This does not mean that the approach does not work. Rather, on the contrary, it is necessary to persistently continue research. It seems to me that this is the right direction, but it may be a task that we solve is actually more complicated than it seemed at the beginning.

As for the role of computer games in the research of AI (and this is chess, GO, Dota-2, and now Starcraft), they are now using it as a convenient training ground, where the game convention and the presence of strict rules help create something working. Real life and any business assaults, of course, are much more complicated. But if you try to imagine which of the real tasks has the greatest chances to benefit from such studies, then I would say that this, of course, is robot training and unmanned driving. It is very promising to combine machine learning approaches and classic optimal control. These are different scientific schools that solve similar problems, but still speak different languages. Their mutual enrichment is quite capable of giving new breakthroughs in the coming years.

Comment. After the publication of the article, the second commentary of the expert was introduced into it.

Alexander Ershov