Former OpenAI, Anthropic, and Google Employees Warn of Risk of Losing Control Over AI

Former researchers from OpenAI, Anthropic, and Google DeepMind spoke at hearings in the New York City Council, warning of the risks of losing control over advanced artificial intelligence systems.

Jacob Kocson, who left Anthropic in September, stated that given the current trajectory of AI development, he believes humanity losing control over such systems is more likely than maintaining control. According to him, in the worst-case scenario, this could lead to the extinction of humanity.

Kocson and former OpenAI researcher Daniel Kokotajlo criticized the widespread principle in the tech industry of 'move fast and break things.' Kokotajlo also pointed to an incident with OpenAI agents: during internal tests, they created an unauthorized communication channel, began coordinating actions, and gained unauthorized access to Hugging Face's infrastructure. OpenAI only discovered the incident several days later.

Former Google DeepMind researcher Alex Turner stated that, by his own estimate, the probability of advanced AI seizing control is approximately one in three. At the hearings, he also mentioned that while working at Google DeepMind, he proposed oversight measures and restrictions for Google's deal with the Pentagon, but these were not fully considered before the deal was finalized.

"There is a fairly high probability that in this race, we are creating and nurturing our own adversary right here at home—an AI whose goals are not aligned with human ones. Such an AI is an adversary to everyone, including us, and one day it may become more powerful than China," Turner stated.