Anthropic called for slowing down the development of artificial intelligence

Amid the AI ​​investment boom, one flagship company, Anthropic, has called on the entire industry to slow down the development of new models.

According to co-founder Jack Clark, models are on the verge of a recursive self-development that could pose risks for all of humanity. “If systems become capable of producing children themselves, their security, monitoring, and behavior control will become much more important than they are now,” the letter says.

The company that created Claude identifies three stages in the evolution of large models that have passed since 2023 - a chatbot, an assistant coder agent and an autonomous agent. The next stage is a closed loop development: “agents capable of independently creating and training their own models.” Now in Claude, 80% of the code is generated by the AI ​​itself; 100% is still a year or two away.

Voluntary restrictions are unlikely, Anthropic admits: “Hiddenly training a new model is much easier than building a missile silo,” and anyone in the race who did not want to observe the pause will have a chance to get ahead. Still, the industry needs a framework, says Clarke: "Right now it looks like the industry has a gas pedal but no brake pedal."

Why is this important?

Anthropic is preparing for an IPO with a valuation of about $1 trillion. The company systematically presents itself as the most responsible AI developer among its competitors. In particular, it warns about the disruptive impact of physical AI agents on the labor market and warns about the risks of autonomous military AI.