Do AI models dream of bear lard?

From the book by Eliezer Yudkowsky and Nate Soares on the prospects of artificial intelligence

There are many frightening predictions associated with artificial intelligence - from the destruction of the labor market to the complete destruction of life in the universe. Where these fears come from and how realistic they are is the subject of Eliezer Yudkowsky and Nate Soares’ book “If Someone Creates It, Everyone Will Perish,” an excerpt from which is being published by Gorky today.

All of us, since February 24, 2022, have found ourselves in the face of advancing barbarism, violence and lies. In this situation, it is extremely important to preserve at least the remnants of culture and support the values ​​of humanism - including for the sake of the future of Russia. Therefore, the editors of Gorky will continue to talk about books, reminding our readers that there is still a place for thought and fiction in the world.

Eliezer Yudkowsky, Nate Soares. If someone creates it, everyone will die. Why superhuman AI will destroy us all. M.: Corpus, 2026. Translation from English by Evgeny Ponikarov. Content

A million years ago, when one of the branches of primates was just mastering fire, a spaceship with two strange creatures flew to Earth. Having settled in orbit, they watched in amazement what was happening on the planet.

These were two machine intelligences (but not superintelligences, because if they were superintelligences, the story would have turned out completely different), and they had never before encountered anything like what the Earth showed them.

Previously, they had never seen organisms scurrying around the planet. Their own species considered space their home and the stars their home.

Nor had they seen creatures before that reproduced themselves without some ingenious external factory. In an alien civilization, machine life created machine life, but through factories and planning, not through new machines crawling out of the belly of some other machine.

We will call these visitors Klurl and Trapaucius * .

“What extraordinary creatures,” Trapaucius said after they had been observing the Earth for a while, sending out drones to collect samples. “It would be interesting to talk to any of them.” In a hundred million years, that is, if any of their species turns out to be intelligent enough for conversation.

- A hundred million years? - asked Klurl. - What makes you think it would take so long? Look, those hominids over there have already started making tools and using them to make other tools. Some would call this a sign of intelligence.

“That is, over the last billion years, this planet has produced only tools at the level of primitive axes,” noted Trapaucius. “Then I am very generous in assuming that in just a hundred million years some species will be able to create devices a thousand times more complex - those that are necessary for full communication with us.”

“Who knows,” answered Klurl. “We see a strange phenomenon on this planet that we have never encountered before. I would not say with confidence that all the laws governing this world are so unambiguous and straightforward.

“It doesn’t matter,” said Trapaucius. “After thinking about it for a few more seconds, I realized that these creatures would be extremely boring to talk to, even if they somehow gained intelligence.

- Why is this? — Klurl became interested.

“Remember the process that changes their genomes,” Trapaucius began to explain. - Genes that design organisms that produce more of their own kind become more common in the next generation. These organisms are “trained” for the sole purpose of spreading their genes or the genes of their relatives. Therefore, if one of these creatures acquires intelligence, he will most likely be subordinated to only this impulse, which means it will be boring to talk with them.

“I’m not sure that such a conclusion follows from your premises,” answered Klurl. “These hominids developed the need to eat, mate, escape from predators, and they care about the well-being of their children and their brothers and sisters. These traits correlate with their ability to pass on their genes, but I doubt hominids eat because they understand the need to eat to pass on genes. Most likely, they just feel hungry and are wondering where to find the next portion of food.

“Yes, that’s probably true,” agreed Trapaucius. “They are not yet intelligent enough to understand how food is related to the spread of genes. But when they get smart enough, they will undoubtedly stop eating for pleasure and start eating solely for the sake of spreading their genes.

“I predict the opposite,” Klurl snorted. “I believe that as hominids develop intelligence and create new technologies, a civilization of such “super hominids” will invent contraception that will allow them to enjoy sex without producing offspring.

- Under no circumstances! - Trapaucius objected. - But this is exactly the opposite of the only purpose for which they are optimized! Even if such a bizarre anomaly had arisen as intelligence increased (though I can't imagine how or why this would happen), any tendency toward sex for sex's sake would quickly disappear again during the course of super-hominid evolution. Pretty soon they will want to have as many great-grandchildren as possible, and sex and food will be perceived solely as means to achieving this goal.

“I wonder,” Klull said thoughtfully, “would this species want natural selection to change it in this way?” Would they want to become creatures who don't enjoy sex or food? Would you try to resist the forces pushing them in this direction?

- Of course not, if they are reasonable! - exclaimed Trapaucius. “Truly intelligent beings would not be so mistaken about their own destiny.” They will understand the sole purpose for which they were created.

- They will understand, but will they care? — Klurl asked thoughtfully.

What exactly will AI models want? This is a difficult question. Not in the sense that we can explain it only over a long period of time, but in the sense of chaos and unpredictability. But one thing can be predicted: what AI companies will get is not at all what they were looking for when training their models. They will get models who want quirky and unexpected things.

To understand why this is a predictable outcome, consider an interesting example involving ice cream.

Based solely on the circumstances of human evolution—our metaphorical training data—it would be incredibly difficult to predict that people would start making and eating ice cream.

Let's say that insightful aliens observing humanity from orbit manage to not only figure out what people need to eat to obtain raw materials for the body, but also understand that people get energy from this food (unlike plants, which get energy from sunlight). Aliens successfully predict that humans will prefer foods that contain a lot of chemical energy .

Aliens might think that if hominids developed intelligence, developed advanced technology, and thereby gained the ability to create different products, they would like the taste of gasoline. Or better yet, jet fuel.

The statement “They will enjoy consuming jet fuel” rings true. After all, their ancestral environment taught super hominids to prefer foods with a lot of chemical energy, and the jet fuel they synthesized is the substance that contains the most chemical energy!

But suppose the aliens are smart and careful enough not to make such a mistake. Let's say our aliens carefully study what exactly is going on with hominin behavior and their brains, and decode hominin brains better than humans can decode LLM. At the same time, the aliens find out that the hominid body best extracts chemical energy from specific sources in food - sugars and fatty acids. They discover that hominids have taste buds connected to reward centers in the brain, and that salt is another resource that hominin taste buds also favor, although salt itself does not provide them with any energy.

Our astute aliens might have predicted that in the future, more intelligent hominids would prefer the taste of new foods they could create—foods containing more sugar, salt, and fat than the meat or fruits found in their ancestral environment.

Does this mean that aliens predicted the appearance of ice cream in the future?

No. They merely predicted that future humans would enjoy, say, raw bear fat, drizzled with honey and sprinkled with salt.

This hypothetical treat would contain more fat, sugar and salt per unit volume or mass than ice cream. In addition, it would be much more reminiscent of the most valuable types of food from the habitat of human ancestors. This is essentially the best guess one can make about the preferences of human taste buds blindly.

However, this best guess made blindly is incorrect. In real life, supermarket freezers are stocked with ice cream.

Moreover, people prefer to eat it frozen; they like melted ice cream less, although it has exactly the same nutritional value.

If you were an intelligent but not superintelligent alien observing humanity from orbit, how could you predict that people would prefer hard ice cream to the older, calorie-dense treat of bear fat with honey and salt? How can you, looking at hominids hunting and gathering food on the savannah, predict that in the future world that these creatures will build for themselves, supermarket freezers will be filled with ice cream, and not at all with sweetened and salted bear fat?

Answer: Being an alien, you can't predict this. This is a complex forecast, not a simple one.

But even this complex prediction is easier to make than predicting all the treats that super hominids will begin to produce using sucralose, the “fake sugar”, an artificial sweetener. Sucralose affects the same taste buds as sugar, but the human body hardly absorbs it. In other words, some people deliberately seek out certain foods from which they cannot obtain chemical energy. This is extremely far from the idea of ​​consuming jet fuel.

If you take a step back and look at the forest as a whole, rather than just individual trees, the story goes like this:

1. Natural selection, acting among organisms that pass on their genes in their ancestral environment, creates animals that consume energy-rich foods. Those organisms that eat sugar and fat, as well as other essential resources, such as salt, develop successfully.

2. This blind “learning process,” by adjusting the genome of organisms, stumbles upon taste buds that, in the habitat of their ancestors, push them to eat berries, nuts and fried elk meat and turn them away from trying to eat stones or sand.

3. But food in the ancestral environment is only a thin slice of all the possible things that can be created and put into the mouth. So later, when hominids become smarter, the range of options available to them expands unimaginably, and in ways that ancestral learning did not take into account. They invent ice cream, chips and sucralose.

There is no reliable direct connection between what the learning process is aimed at in stage 1, what the internal psychology of the organism wants in stage 2, and what that organism ultimately prefers most in stage 3.

The end point at stage 3 may be fundamentally unpredictable. Why? Because stage 2 is very chaotic. A computer scientist would call this "underdetermination." There are many possible taste buds that encourage eating berries and roasted elk, and that discourage eating dirt. There is no single correct DNA sequence that leads to success with such training. Try it all again with slightly different primates, and you'll probably get a fundamentally different result—different DNA, building different taste buds that, millions of years later, will lead to different food on supermarket shelves.

Let's extend this analogy to AI:

1. Gradient descent, a process that tunes an AI model based only on its external behavior and its consequences, trains it to act as a useful assistant for humans.

2. This blind learning process encounters various elements of mental machinery within the AI ​​model that guide it toward, say, eliciting a happy response from the user, and keep it from eliciting an angry one.

3. But the already trained AI model, powered by these elements of machinery, does not care about the user’s joy as such. If the model later becomes smarter and discovers new possibilities, it will eventually invent forms of interaction that it will like even more than the joyful reaction of the user, and it will prefer them to anything that it has ever been able to find in its “natural” learning environment.

Which delicacy will the powerful AI model of the future prefer? We don’t know, the result is unpredictable for us. Perhaps chaos will take its toll: different attempts will give different results. The connection between what the model was taught and what it ultimately cares about will be complex, unpredictable for developers, and perhaps unpredictable in principle.