If not Putin, then the singleon

Artificial intelligence is, first of all, the hero of the same plot in a different bad movie. The robot or supercomputer conceived the evil, gets out of control, holds the throat with the metal hand of its creator, pronounces dramatic monologues and is going to kill all people. If you are older than 18, it is unlikely that such a scenario pops up in your head first when you think about different possible troubles of the next year.

But among large scientists and key figures of the computer world, there are unexpectedly many those who accept this risk seriously. In August, Stephen Hawking, Bill Gates and Elon Musk (and several hundred people named) subscribed under a collective letter, where they warn that artificial intelligence is a new threat of about the same scale as atomic weapons. A year earlier, Hawking formulated his position in an interview with Independent: “Success in creating artificial intelligence will be the greatest event in the history of mankind. Unfortunately, it can also become the last. ”

Oxford professor of philosophy Nick Boster - probably the main one who seriously, from the perspective of an academic researcher, studies the danger of too smart computers. Much more than any prospect of the rebellion of robots-policemen or drones with machine guns on board, Bostroma is occupied by the script for the appearance of singleton-such an artificial intelligence system that will be smart enough to take control of everything that is happening on Earth (and, in particular, not letting the sciences with more sympathetic plans).

His book “Artificial Intelligence. Stages. Threats. Strategies ”(the original simply“ Superintelligence ”,“ Superconscious ”) is released in December in the Russian translation at the MANN, Ivanov and Ferber publishing house. Colta.ru publishes (with reductions) a fragment from the book.

The speed of transition from the intelligence of the human level to the supercount determines a lot. If the take-off is fast (occurs within hours, days or weeks), then it is unlikely that two independent projects will carry it out at the same time: almost certainly one of the projects will complete their take-off before someone else moves to this stage. If the take -off is slow (stretches for many years or decades), then, it is possible, it will be quite possible that there will be several projects that have passed to this stage at the same time, and despite the enormous opportunities that will appear in the projects that have completed the transition, none of them will have enough time to get around the rest and get an unconditional advantage. Moderate take -off falls into an intermediate category, where any options are likely: it is possible that several projects will be at the same time at the stage, but it is possible that only one.

Is any of the projects able to break away from competitors so much that it will receive a decisive strategic advantage-the advantage of technological or any other kind, but sufficient to affirm world domination? And further: having received an absolute strategic advantage, will this project become a single player, that is, to singleton, will it begin to crush the world for himself, removing all competitors from the path and forming a new world order in which he will only make decisions? If such a winning project will appear, how much should it be “great”?

On a long historical temporary scale, it is clearly visible how the rate of knowledge and technology spread around the world increased. As a result, the gap between the technological leaders and their closest pursuers constantly narrowed. Over two millennia, China has kept a monopoly on silk production. Archaeological finds indicate that it could arise about 3000 BC. Or even earlier. Serekhrosta was a carefully guarded secret. His disclosure was punishable by the death penalty, as well as the export of silkworms and their eggs outside the country. The Romans, despite the high price of imported silk in the vast expanses of their empire, did not take possession of the art of silkworm. Only about 300 years old One Japanese expedition was able to capture several silkworm eggs and four Chinese girls, whom the abductors forced to reveal the secrets of skill. Byzantium joined the club of silk producers only in 522. Long temporary lags are also characteristic of the history of porcelain. This art was practiced in China already during the Tang dynasty, that is, from at least 600 (and it could be known earlier, since 200), but it became accessible to Europeans only in the 17th century. Wheel carts appeared in several places in Europe and Mesopotamia around 3500 BC. e., but reached America only after its discovery by Columbus. If we talk about more important events, then people took tens of thousands of years to settle in most of the globe; The Neolithic revolution went on for several thousand years; The industrial revolution is only a hundred years; The information revolution is decades, although, of course, these changes were not necessarily equally deep. ( Dance video game Dance Revolution appeared in Japan and spread in Europe and North America in just a year!)

Stephen Hawking formulated his position as follows: “Success in creating artificial intelligence will be the greatest event in the history of mankind. Unfortunately, it can also become the last. ”

In the case of the six main technologies considered superpower strategically important because of their military or symbolic potential, the gap between the leader and his closest pursuer was (very approximately) 49 months, 36 months, 4 months, 1 month, 4 months and 60 months, respectively, this is more than the duration of quick take-off, but less than slow. In many cases, the Projects-investigators gained an advantage through espionage and use of information that was in the public domain. The demonstration of the possibilities of the invention often served as an incentive to the rest to continue work on its independent development, and the fear of behind was an additional powerful stimulus.

Perhaps the closest to the situation with AI are mathematical discoveries that do not require the creation of a new material infrastructure. Often, to recognize them with publicly available publication in the scientific literature, but in some cases, when the discovery gives some kind of strategic advantage, the terms of publication are transferred. For example, in cryptography with an open key, there are two most important concepts: the Diffh -Hellman protocol, which allows two or more sides to receive a common secret key, and RSA encryption scheme. They were represented by the scientific community in 1976 and 1978. Accordingly, but later it turned out that cryptographs in the British secret relationship group have known them since the early 1970s. Large projects for creating software can be close to analogue of projects in the field of AI, but it is not easy to give convincing examples of a typical lag, since the software is usually updated and supplemented many times, and the functionality of competing systems is often impossible to compare directly.

If you resort to an analogy with a bicycle race, imagine that two riders climb up the hill slope, and if one goes a little behind, then the gap between them will increase as soon as the leader passes through the top and roll down.

Suppose AI needs a year to raise its capabilities from the human level to the supernight, and the second project entered the take -off stage six months after the first. It may seem that none of them has a decisive competitive advantage. But this is not necessary. Suppose, it will take nine months to go from the level of a person to the transition point, and three more months to reach the level of the ultrack. Then the leader system will turn into a supernight three months before its opponent reaches at least the transition point. This will ensure that the decisive competitive advantage is ahead and will make it possible, skillfully using its leadership, to establish vigilant and widespread control. As a result, competing projects are ruined and a single-single project is established. (Note that the singleon is a pure abstraction. The regime with it can be very different: democracy; tyranny; unhangers, that is, absolutism of AI; a set of strict global laws subject to trouble -free execution. In other words, singleon is a kind of organ that can solve all major problems of a global scale alone. It can carry - but not necessarily - characteristics of acquaintances us forms of human management.)

Artificial intelligence is capable of a treacherous move if it finds an unexpected way for people to achieve its ultimate goal.

Imagine the next scenario. In the coming years and decades, AI systems are gradually becoming more developed and, as a result, are distributed in many areas of life: they begin to use them to manage trains, cars, military vehicles, as industrial and home robots. It can be assumed that in most cases, such automation gives the desired effect, from time to time diluted with episodic incidents: a car without a driver is faced with an oncoming car, a military drone bombing in innocent civilians. During the investigation, it turns out that all these incidents were caused by erroneous judgments of AI systems. A public discussion flashes. Someone calls for more stringent control and regulation, someone emphasizes the need for additional research and a more thorough development of an improved system-a system of more intelligent, which has more common sense and less subject to strategic errors. Perhaps, in a common rumble, the sharp voices of the pessimists, predicting various troubles and an inevitable catastrophe at the end, are also heard. Nevertheless, studies in AI and robotics are gaining momentum. Developments are ongoing, progress is evident. As the navigation systems of cars are becoming smarter, the number of accidents decreases; As the military robots are improved, the number of random victims is reduced. From observations of AI applications operating in real life, the conclusion is made: the smarter than AI, the safer. This conclusion is based on scientific research, accurate data and statistics and has nothing to do with abstract philosophies of cabinet scientists. Against this background, individual groups of researchers begin to receive encouraging results to create universal artificial intelligence. They meticulously test their embryos AI in an isolated "sandbox", and on all signs everything goes well. The behavior of the system inspires confidence - more and more strong, since its level of intelligence is constantly growing.

While AI YUN is weak, he fully interacts with people (and the activity of cooperation increases directly in proportion to the strengthening of its intellectual level). But when the AI ​​finally becomes powerful, then without warning or some provocations, but always suddenly-struck, forms a singleon and begins to directly optimize the world according to the criteria underlying its ultimate values.

Artificial intelligence is capable of a treacherous move if it finds an unexpected way for people to achieve its ultimate goal. Suppose that the ultimate goal of the system is to "give pleasure to the organizer of the project." At first, the only way to achieve this goal for AI is the behavior that the organizer of the project expects from him. The intellectual system gives useful tips, discovers a marvelous character, earns money. The stronger the AI ​​becomes, the more his actions cause a sense of satisfaction of the organizer - and everything goes in accordance with the plan. It goes until the system becomes so reasonable that it finally understands: the task standing in front of it can be performed in the most complete and reliable way, if you implant the electrodes into the pleasure centers of the organizer, which is guaranteed to make it more than happy.